egeria-server-config.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Egeria Logo](https://raw.githubusercontent.com/odpi/egeria/main/assets/img/ODPi_Egeria_Logo_color.png)\n",
    "\n",
    "### Egeria Hands-On Lab\n",
    "# Welcome to the Configuring Egeria Servers Lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Egeria is an open source project that provides open standards and implementation libraries to connect tools, catalogs and platforms together so they can share information about data and technology.  This information is called metadata.\n",
    "\n",
    "In this hands-on lab you will learn how to configure the metadata servers used by [Coco Pharmaceuticals](https://egeria-project.org/practices/coco-pharmaceuticals/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The scenario\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/odpi/egeria-docs/main/site/docs/practices/coco-pharmaceuticals/personas/gary-geeke.png\" style=\"float:left\">\n",
    "\n",
    "Coco Pharmaceuticals is going through a major business transformation that requires them to drastically reduce their cycle times, collaborate laterally across the different parts of the business and react quickly to the changing needs of their customers. (See [this link](https://egeria-project.org/practices/coco-pharmaceuticals/) for the background to this transformation).\n",
    "\n",
    "Part of the changes needed to the IT systems that support the business is the roll out of a distributed open metadata and governance capability that is provided by Egeria.\n",
    "\n",
    "[Gary Geeke](https://egeria-project.org/practices/coco-pharmaceuticals/personas/gary-geeke/) is the IT Infrastructure leader at Coco Pharmaceuticals.\n",
    "\n",
    "In this hands-on lab Gary is configuring the servers that support this open ecosystem.  These servers are collectively called Open Metadata and Governance (OMAG) Servers.\n",
    "\n",
    "Gary's userId is `garygeeke`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "adminUserId     = \"garygeeke\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "He needs to define the OMAG servers for Coco Pharmaceuticals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "organizationName = \"Coco Pharmaceuticals\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open Metadata and Governance (OMAG) management landscape\n",
    "\n",
    "At the heart of an open metadata and governance landscape are the servers that store and exchange metadata in a peer-to-peer exchange called the\n",
    "[open metadata repository cohort](https://egeria-project.org/concepts/cohort-member/).\n",
    "These servers are collectively called **cohort members**.  There are three types of cohort member that Gary needs to consider:\n",
    "\n",
    "* A [Metadata Access Store](https://egeria-project.org/concepts/metadata-access-store/) that uses\n",
    "  a native Egeria repository to store open metadata.  There should be at least one of these servers in a cohort. It used to support\n",
    "  either a community of users that are using the Egeria functions directly or to fill in any gaps in the metadata support provided by the\n",
    "  third party tools that are connected to the cohort.\n",
    "  \n",
    "* A [Metadata Access Point](https://egeria-project.org/concepts/metadata-access-point) that \n",
    "  has no metadata repository of its own and uses federated queries to retrieve and store metadata in the other repositories connected to the cohort.\n",
    "  \n",
    "* A [Repository Proxy](https://egeria-project.org/concepts/repository-proxy) that connects in a third party metadata server.\n",
    "\n",
    "Gary has decided to deploy a separate cohort member server for each part of the organization that owns \n",
    "[assets](https://egeria.odpi.org/open-metadata-implementation/access-services/docs/concepts/assets/).\n",
    "You can think of each of these servers as supporting a community of users within Coco Pharmaceuticals.  The servers are as follows:\n",
    "\n",
    "* cocoMDS1 - Data Lake Operations - a **metadata access store** used to manage the data in the data lake.\n",
    "* cocoMDS2 - Governance - a **metadata access store** used by all of the governance teams to operate the governance programs.\n",
    "* cocoMDS3 - Research - a **metadata access store** used by the research teams who are developing new treatments.\n",
    "* cocoMDS4 - Data Lake Users - a **metadata access point** used by general business users and the executive team to access data\n",
    "  from the data lake.\n",
    "* cocoMDS5 - Business Systems - a **repository proxy** used to connect to the existing ETL tool that manages data movement amongst the\n",
    "  business systems.  It has a metadata record of the operational business systems such as procurements, sales, human resources and\n",
    "  finance and the movement of data between them.  This tool is also loading data from the business systems into the data lake. Its metadata\n",
    "  is critical for providing lineage for the data used to run the business.\n",
    "* cocoMDS6 - Manufacturing - a **metadata access store** used by the supplies warehouse, manufacturing and distribution teams.\n",
    "* cocoMDSx - Development - a **metadata access store** used by the software development teams building new IT capability.\n",
    "* cocoEDGEi - Manufacturing sensors edge node servers (many of them) - these **metadata access stores** catalog the collected sensor data.\n",
    "\n",
    "In addition, Coco Pharmaceuticals needs additional servers to support Egeria's user interface and automated metadata processing:\n",
    "\n",
    "* cocoView1 - a [View Server](https://egeria-project.org/concepts/view-server) \n",
    "  that runs the services for the user interface.           \n",
    "* exchangeDL01 - an [Integration Daemon](https://egeria-project.org/concepts/integration-daemon) \n",
    "  server that supports the automatic exchange of metadata with third party technologies.          \n",
    "* governDL01 - an [Engine Host](https://egeria-project.org/concepts/engine-host)\n",
    "  server that runs governance functions that monitor, validate, correct and enrich metadata for use by all of the technologies in the connected\n",
    "  open metadata ecosystem.\n",
    "* cocoOLS1 - an [Open Lineage Server](https://egeria-project.org/concepts/open-lineage-server) that manages a historical warehouse of lineage information.\n",
    "\n",
    "These servers will each be configured in later parts of this hands-on lab, but first there are decisions to be made about the platform that the servers will run on and how they will be connected together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Metadata and Governance (OMAG) Server Platforms\n",
    "\n",
    "Coco Pharmaceuticals' servers must be hosted on at least one OMAG Server Platform.\n",
    "This is a single executable (application) that can be started from the command line or a script or as part of a \n",
    "pre-built container environment `kubernetes`.\n",
    "\n",
    "If you are running this notebook as part of an Egeria hands on lab then the server platforms you need are already started.  Run the following command to check that the  platforms are running.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run common/common-functions.ipynb\n",
    "%run common/environment-check.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "If one of the platforms is not running, follow [this link to set up and run the platform](https://egeria-project.org/education/open-metadata-labs/overview/).  Once the platforms are running you are ready to proceed.\n",
    "\n",
    "----\n",
    "Most of the servers are supporting a pretty stable environment and can share an OMAG Server Platform because the workload they are supporting is predicable.\n",
    "The data lake however requires a lot of active governance and is evolving rapidly.\n",
    "\n",
    "To isolate this churn, Gary chooses to put all of the metadata and governance servers for the data lake on to their own platform.\n",
    "The development team requested that their infrastructure is completely separate from the operational systems,\n",
    "so they are given their own server platform.\n",
    "Finally each of the edge servers will run their own OMAG Server Platform to support their own metadata server.\n",
    "\n",
    "Figure 1 shows which servers will sit in each platform.  The cohort members are shown in white,\n",
    "governance servers in orange and the view server (that supports the UI) is in green.\n",
    "\n",
    "![Figure 1](images/coco-pharmaceuticals-systems-omag-server-platforms.png)\n",
    "> **Figure 1:** Coco Pharmaceuticals' OMAG Server Platforms\n",
    "\n",
    "\n",
    "The sensor edge node servers used to monitor the warehouse operation and manufacturing process each have their own platform and are not yet included in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Metadata Repository Cohorts\n",
    "\n",
    "A metadata server, metadata access point and repository proxy can become a member of none, one or many cohorts.\n",
    "Once a server has joined a cohort it can exchange metadata with the other members of that cohort.\n",
    "So the cohorts define scopes of sharing.\n",
    "\n",
    "Gary decides to begin with three open metadata repository cohorts:\n",
    "\n",
    "* **cocoCohort** - The production cohort contains all of the servers that are used to run, coordinate and govern the business.\n",
    "* **devCohort** - The development cohort where the development teams are building and testing new capability.  Much of their metadata describes the software components under construction and the governance of the software development lifecycle.\n",
    "* **iotCohort** - The IoT cohort used to manage the sensors and robots in the manufacturing systems.  The metadata produced by the sensors and robots is only of interest to the manufacturing and governance team.\n",
    "\n",
    "Figure 2 shows which servers belong to each cohort.\n",
    "\n",
    "![Figure 2](images/coco-pharmaceuticals-systems-cohorts.png)\n",
    "> **Figure 2:** Membership of Coco Pharmaceuticals' cohorts\n",
    "\n",
    "Below are the names of the three cohorts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cocoCohort = \"cocoCohort\"\n",
    "devCohort  = \"devCohort\"\n",
    "iotCohort  = \"iotCohort\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the heart of each cohort is an event topic.  By default, Egeria uses [Apache Kafka](https://kafka.apache.org/) topics.\n",
    "The servers that will join a cohort need to be configured with the host name and port for Kafka.\n",
    "The command below pulls the value from an environment variable called `eventBusURLroot` with a default value of\n",
    "`localhost:9092`.  It is used in all of the server configuration documents to connect it to Kafka."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eventBusURLroot   = os.environ.get('eventBusURLroot', 'localhost:9092')\n",
    "\n",
    "eventBusBody      = {\n",
    "    \"producer\": {\n",
    "        \"bootstrap.servers\": eventBusURLroot\n",
    "    },\n",
    "    \"consumer\":{\n",
    "        \"bootstrap.servers\": eventBusURLroot\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access services\n",
    "\n",
    "[The Open Metadata Access Services (OMAS)](https://egeria-project.org/services/omas/) provide domain-specific services for data tools, engines and platforms to integrate with open metadata. These are the different types of access service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "getAccessServices(cocoMDS1PlatformName, cocoMDS1PlatformURL)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The table below shows which access services are needed by each server.\n",
    "\n",
    "\n",
    "| Access Service       | cocoMDS1 | cocoMDS2 | cocoMDS3 | cocoMDS4 | cocoMDS5 | cocoMDS6 | cocoMDSx | cocoEDGE*i* |\n",
    "| :------------------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :---------: |\n",
    "| asset-catalog        |   Yes    |   Yes    |   Yes    |   Yes    |   No     |   Yes    |   Yes    |     No      |\n",
    "| asset-consumer       |   Yes    |   Yes    |   Yes    |   Yes    |   No     |   Yes    |   Yes    |     No      |\n",
    "| asset-owner          |   Yes    |   Yes    |   Yes    |   No     |   No     |   Yes    |   Yes    |     No      |\n",
    "| community-profile    |   Yes    |   Yes    |   Yes    |   Yes    |   No     |   Yes    |   Yes    |     No      |\n",
    "| glossary-view        |   Yes    |   Yes    |   Yes    |   Yes    |   No     |   Yes    |   Yes    |     No      |\n",
    "| -------------------  |  ------  |  ------  | ------   |  ------  |  ------  |  ------  |  ------  | ----------  |\n",
    "| data-science         |   No     |   No     |   Yes    |   Yes    |   No     |   Yes    |   Yes    |     No      |\n",
    "| -------------------  |  ------  |  ------  | ------   |  ------  |  ------  |  ------  |  ------  | ----------  |\n",
    "| subject-area         |   No     |   Yes    |   Yes    |   No     |   No     |   Yes    |   Yes    |     No      |\n",
    "| -------------------  |  ------  |  ------  | ------   |  ------  |  ------  |  ------  |  ------  | ----------  |\n",
    "| governance-program   |   No     |   Yes    |   No     |   No     |   No     |   No     |   No     |     No      |\n",
    "| data-privacy         |   No     |   Yes    |   No     |   No     |   No     |   No     |   No     |     No      |\n",
    "| security-officer     |   No     |   Yes    |   No     |   No     |   No     |   No     |   No     |     No      |\n",
    "| asset-lineage        |   No     |   Yes    |   No     |   No     |   No     |   No     |   No     |     No      |\n",
    "| -------------------- |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  | ----------  |\n",
    "| discovery-engine     |   Yes    |   No     |   Yes    |   No     |   No     |   Yes    |   Yes    |     No      |\n",
    "| governance-engine    |   Yes    |   Yes    |   Yes    |   No     |   No     |   Yes    |   Yes    |     No      |\n",
    "| asset-manager        |   Yes    |   No     |   Yes    |   No     |   No     |   Yes    |   Yes    |     No      |\n",
    "| -------------------- |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  | ----------  |\n",
    "| data-engine          |   Yes    |   No     |   No     |   No     |   No     |   Yes    |   No     |     Yes     |\n",
    "| data-manager         |   Yes    |   No     |   No     |   No     |   No     |   Yes    |   Yes    |     Yes     |\n",
    "| -------------------- |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  | ----------  |\n",
    "| it-infrastructure    |   No     |   Yes    |   No     |   No     |   No     |   Yes    |   Yes    |     No      |\n",
    "| project-management   |   No     |   Yes    |   Yes    |   No     |   No     |   Yes    |   Yes    |     No      |\n",
    "| -------------------- |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  | ----------  |\n",
    "| software-developer   |   No     |   No     |   No     |   No     |   No     |   No     |   Yes    |     No      |\n",
    "| devops               |   No     |   No     |   No     |   No     |   No     |   No     |   Yes    |     No      |\n",
    "| digital-architecture |   Yes    |   Yes    |   No     |   No     |   No     |   No     |   Yes    |     No      |\n",
    "| design-model         |   No     |   No     |   No     |   No     |   No     |   No     |   Yes    |     No      |\n",
    "| -------------------- |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  |  ------  | ----------  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Egeria Server Configuration Overview\n",
    "\n",
    "Open metadata servers are configured using REST API calls to an OMAG Server Platform.  Each call either defines a default value or configures a service that must run within the server when it is started.\n",
    "\n",
    "As each configuration call is made, the OMAG Server Platform builds up a [configuration document](https://egeria-project.org/concepts/configuration-document) with the values passed.  When the configuration is finished, the configuration document will have all of the information needed to start the server.\n",
    "\n",
    "The configuration document will then be deployed with the OMAG Server Platform that is to host the server.  When a request is made to this OMAG Server Platform to start the server, it reads the configuration document and initializes the server with the appropriate services.\n",
    "\n",
    "## Configuration Set Up\n",
    "\n",
    "A server can be configured by any OMAG Server Platform - it does not have to be the same platform where the server will run.  For this hands on lab we will use the development team's OMAG Server Platform to create the servers' configuration documents and then deploy them to the platforms where they will run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "adminPlatformURL = devPlatformURL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The URLs for the configuration REST APIs have a common structure and begin with the following root:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adminCommandURLRoot = adminPlatformURL + '/open-metadata/admin-services/users/' + adminUserId + '/servers/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of Coco Pharmaceuticals' metadata servers need a local repository to store metadata about the data and processing occurring in the data lake.\n",
    "\n",
    "Egeria includes two types of repositories natively. One is an **in-memory repository** that stores metadata in hash maps. It is useful for demos and testing because a restart of the server results in an empty metadata repository. However, if you need metadata to persist from one run of the server to the next, you should use the **local graph repository**.\n",
    "\n",
    "The choice of local repository is made by specifying the local repository mode. The variables below show the two options. The `metadataRepositoryType` identifies which one is going to be used in the configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inMemoryRepositoryOption = \"in-memory-repository\"\n",
    "graphRepositoryOption    = \"local-graph-repository\"\n",
    "\n",
    "# Pick up which repo type to use from environment if set, otherwise default to inmemory \n",
    "metadataRepositoryType   = os.environ.get('repositoryType', inMemoryRepositoryOption)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Egeria supports instance based security.  These checks can be customized through an\n",
    "[Server Metadata Security Connector](https://egeria-project.org/concepts/server-metadata-security-connector/).\n",
    "Coco Pharmaceuticals have written their own connector to support the specific rules of their industry.\n",
    "The Connection definition below tells a server how to load this connector.  It needs to be included in each server's configuration document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "serverSecurityConnectionBody = {\n",
    "    \"class\": \"Connection\",\n",
    "    \"connectorType\": {\n",
    "        \"class\": \"ConnectorType\",\n",
    "        \"connectorProviderClassName\": \"org.odpi.openmetadata.metadatasecurity.samples.CocoPharmaServerSecurityProvider\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, to ensure that a caller can not request too much metadata in a single request, it is possible to set a maximum page size for requests that return a list of items.  The maximum page size puts a limit on the number of items that can be requested.  The variable below defines the value that will be added to the configuration document for each server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "maxPageSize = '600'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring cocoMDS1 - Data Lake Operations metadata server\n",
    "\n",
    "This section configures the `cocoMDS1` server.  The server name is passed on every configuration call to identify which configuration document to update with the new configuration.  The configuration document is created automatically on first use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdrServerName          = \"cocoMDS1\"\n",
    "mdrServerUserId        = \"cocoMDS1npa\"\n",
    "mdrServerPassword      = \"cocoMDS1passw0rd\"\n",
    "mdrServerPlatform      = dataLakePlatformURL\n",
    "metadataCollectionName = \"Data Lake Catalog\"\n",
    "\n",
    "print(\"Configuring \" + mdrServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, mdrServerName, mdrServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, mdrServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, mdrServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, mdrServerName, mdrServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, mdrServerName, mdrServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, mdrServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, mdrServerName, eventBusBody)\n",
    "configureMetadataRepository(adminPlatformURL, adminUserId, mdrServerName, metadataRepositoryType)\n",
    "configureDescriptiveName(adminPlatformURL, adminUserId, mdrServerName, metadataCollectionName)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, cocoCohort)\n",
    "\n",
    "print(\"\\nConfiguring \" + mdrServerName + \" Access Services (OMAS)...\")\n",
    "\n",
    "accessServiceOptions = {\n",
    "    \"SupportedZones\": [\"quarantine\", \"clinical-trials\", \"research\", \"data-lake\", \"trash-can\"]\n",
    "}\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-catalog', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-consumer', accessServiceOptions)\n",
    "\n",
    "accessServiceOptions[\"DefaultZones\"] = [ \"quarantine\" ]\n",
    "accessServiceOptions[\"PublishZones\"] = [ \"data-lake\" ]\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-manager', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-owner', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'community-profile', {\"KarmaPointPlateau\":\"500\"})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'glossary-view', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'discovery-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-manager', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'digital-architecture', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'governance-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-lineage', accessServiceOptions)\n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## Configuring cocoMDS2 - Governance metadata server\n",
    "\n",
    "This section configures the `cocoMDS2` server.  This server is configured in a similar way to cocoMDS1 except that is has different Open Metadata Access Services (OMASs) enabled and it joins all of the cohorts.\n",
    "\n",
    "The code below covers the basic set up of the server properties, security, event bus and local repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdrServerName          = \"cocoMDS2\"\n",
    "mdrServerUserId        = \"cocoMDS2npa\"\n",
    "mdrServerPassword      = \"cocoMDS2passw0rd\"\n",
    "mdrServerPlatform      = corePlatformURL\n",
    "metadataCollectionName = \"Governance Catalog\"\n",
    "\n",
    "print(\"Configuring \" + mdrServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, mdrServerName, mdrServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, mdrServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, mdrServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, mdrServerName, mdrServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, mdrServerName, mdrServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, mdrServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, mdrServerName, eventBusBody)\n",
    "configureMetadataRepository(adminPlatformURL, adminUserId, mdrServerName, metadataRepositoryType)\n",
    "configureDescriptiveName(adminPlatformURL, adminUserId, mdrServerName, metadataCollectionName)\n",
    "# Note: cohort membership is configured for all of the cohorts here\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, cocoCohort)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, devCohort)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, iotCohort)\n",
    "\n",
    "print(\"\\nConfiguring \" + mdrServerName + \" Access Services (OMAS)...\")\n",
    "\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-catalog', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-consumer', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-owner', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'community-profile', {\"KarmaPointPlateau\":\"500\"})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'glossary-view', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'subject-area', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'governance-engine', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'governance-program', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-privacy', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'digital-architecture', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'security-officer', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-lineage', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'it-infrastructure', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'project-management', {})\n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## Configuring cocoMDS3 - Research\n",
    "\n",
    "Server cocoMDS3 is used by the research teams who are developing new treatments.\n",
    "These teams are working with their own assets as well as assets coming from the data lake.\n",
    "So they have their own repository and connector to the core cohort to access all of the\n",
    "operational metadata.\n",
    "\n",
    "This is one of the big changes brought by Coco Pharmaceuticals' business transformation.\n",
    "In their old business model, the research teams were completely separate from the operational\n",
    "part of the organization.  Now they need to be an active member of the day to day running of\n",
    "the organization, supporting the development of personalized medicines and their use in\n",
    "treating patients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdrServerName          = \"cocoMDS3\"\n",
    "mdrServerUserId        = \"cocoMDS3npa\"\n",
    "mdrServerPassword      = \"cocoMDS3passw0rd\"\n",
    "mdrServerPlatform      = corePlatformURL\n",
    "metadataCollectionName = \"Research Catalog\"\n",
    "\n",
    "print(\"Configuring \" + mdrServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, mdrServerName, mdrServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, mdrServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, mdrServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, mdrServerName, mdrServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, mdrServerName, mdrServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, mdrServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, mdrServerName, eventBusBody)\n",
    "configureMetadataRepository(adminPlatformURL, adminUserId, mdrServerName, metadataRepositoryType)\n",
    "configureDescriptiveName(adminPlatformURL, adminUserId, mdrServerName, metadataCollectionName)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, cocoCohort)\n",
    "\n",
    "print(\"\\nConfiguring \" + mdrServerName + \" Access Services (OMAS)...\")\n",
    "\n",
    "accessServiceOptions = {\n",
    "    \"SupportedZones\": [\"personal-files\", \"clinical-trials\", \"research\", \"data-lake\", \"trash-can\"]\n",
    "}\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-catalog', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-consumer', accessServiceOptions)\n",
    "\n",
    "accessServiceOptions[\"DefaultZones\"] = [ \"personal-files\" ]\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-owner', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'community-profile', {\"KarmaPointPlateau\":\"500\"})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'glossary-view', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-science', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'subject-area', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-manager', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'governance-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'discovery-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'project-management', accessServiceOptions)\n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Configuring cocoMDS4 - Data Lake Users\n",
    "\n",
    "Server cocoMDS4 used by general business users and the executive team to access data from the data lake.\n",
    "It does not have a repository of its own.  Instead it issues federated queries to the other repositories in the `cocoCohort`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdrServerName          = \"cocoMDS4\"\n",
    "mdrServerUserId        = \"cocoMDS4npa\"\n",
    "mdrServerPassword      = \"cocoMDS4passw0rd\"\n",
    "mdrServerPlatform      = dataLakePlatformURL\n",
    "metadataCollectionName = \"Data Lake Catalog\"\n",
    "\n",
    "print(\"Configuring \" + mdrServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, mdrServerName, mdrServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, mdrServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, mdrServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, mdrServerName, mdrServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, mdrServerName, mdrServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, mdrServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, mdrServerName, eventBusBody)\n",
    "# Note: no metadata repository or collection configuration here\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, cocoCohort)\n",
    "\n",
    "print(\"\\nConfiguring \" + mdrServerName + \" Access Services (OMAS)...\")\n",
    "\n",
    "accessServiceOptions = {\n",
    "    \"SupportedZones\": [ \"data-lake\" ]\n",
    "}\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-catalog', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-consumer', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'community-profile', {\"KarmaPointPlateau\":\"500\"})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'glossary-view', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-science', accessServiceOptions)\n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## Configuring cocoMDS5 - Business Systems\n",
    "\n",
    "Server cocoMDS5 is a repository proxy to an ETL tool called `iisCore01`.  This ETL tool is well established in Coco Pharmaceuticals and has a built-in metadata repository that contains information about their operational business systems such as procurement, sales, human resources and finance.\n",
    "\n",
    "This ETL tool has its own user interface and services so the OMASs are not enabled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdrServerName          = \"cocoMDS5\"\n",
    "mdrServerUserId        = \"cocoMDS5npa\"\n",
    "mdrServerPassword      = \"cocoMDS5passw0rd\"\n",
    "mdrServerPlatform      = corePlatformURL\n",
    "metadataCollectionName = \"Business Systems Catalog\"\n",
    "\n",
    "print(\"Configuring \" + mdrServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, mdrServerName, mdrServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, mdrServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, mdrServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, mdrServerName, mdrServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, mdrServerName, mdrServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, mdrServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, mdrServerName, eventBusBody)\n",
    "configureRepositoryProxyDetails(adminPlatformURL, adminUserId, mdrServerName, \"org.odpi.openmetadata.adapters.repositoryservices.readonly.repositoryconnector.ReadOnlyOMRSRepositoryConnectorProvider\")\n",
    "configureDescriptiveName(adminPlatformURL, adminUserId, mdrServerName, metadataCollectionName)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, cocoCohort)\n",
    "# Note: no access service configuration here\n",
    "# Still need to add startup Archive \n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## Configuring cocoMDS6 - Manufacturing\n",
    "\n",
    "Server cocoMDS6 is the repository server used by the warehouse, manufacturing and distribution teams.  It supports the systems for this part of the organization and acts as a hub for monitoring the IoT environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdrServerName          = \"cocoMDS6\"\n",
    "mdrServerUserId        = \"cocoMDS6npa\"\n",
    "mdrServerPassword      = \"cocoMDS6passw0rd\"\n",
    "mdrServerPlatform      = corePlatformURL\n",
    "metadataCollectionName = \"Manufacturing Catalog\"\n",
    "\n",
    "print(\"Configuring \" + mdrServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, mdrServerName, mdrServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, mdrServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, mdrServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, mdrServerName, mdrServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, mdrServerName, mdrServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, mdrServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, mdrServerName, eventBusBody)\n",
    "configureMetadataRepository(adminPlatformURL, adminUserId, mdrServerName, metadataRepositoryType)\n",
    "configureDescriptiveName(adminPlatformURL, adminUserId, mdrServerName, metadataCollectionName)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, cocoCohort)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, iotCohort)\n",
    "\n",
    "print(\"\\nConfiguring \" + mdrServerName + \" Access Services (OMAS)...\")\n",
    "\n",
    "accessServiceOptions = {\n",
    "    \"SupportedZones\": [ \"manufacturing\" ],\n",
    "    \"DefaultZones\" : [ \"manufacturing\"]\n",
    "}\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-catalog', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-consumer', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-owner', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'community-profile', {\"KarmaPointPlateau\":\"500\"})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'glossary-view', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-science', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'subject-area', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-manager', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'governance-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'discovery-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-manager', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'governance-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'it-infrastructure', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'project-management', accessServiceOptions)\n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## Configuring cocoMDSx - Development\n",
    "\n",
    "Server cocoMDSx is used by the development teams building new IT capability.  It will hold all of the software component assets and servers used for development and devOps.  The development teams have their own OMAG Server Platform and cohort called 'devCohort'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdrServerName          = \"cocoMDSx\"\n",
    "mdrServerUserId        = \"cocoMDSxnpa\"\n",
    "mdrServerPassword      = \"cocoMDSxpassw0rd\"\n",
    "mdrServerPlatform      = devPlatformURL\n",
    "metadataCollectionName = \"Development Catalog\"\n",
    "\n",
    "print(\"Configuring \" + mdrServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, mdrServerName, mdrServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, mdrServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, mdrServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, mdrServerName, mdrServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, mdrServerName, mdrServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, mdrServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, mdrServerName, eventBusBody)\n",
    "configureMetadataRepository(adminPlatformURL, adminUserId, mdrServerName, metadataRepositoryType)\n",
    "configureDescriptiveName(adminPlatformURL, adminUserId, mdrServerName, metadataCollectionName)\n",
    "configureCohortMembership(adminPlatformURL, adminUserId, mdrServerName, devCohort)\n",
    "\n",
    "print(\"\\nConfiguring \" + mdrServerName + \" Access Services (OMAS)...\")\n",
    "\n",
    "accessServiceOptions = {\n",
    "    \"SupportedZones\": [ \"sdlc\", \"quarantine\", \"clinical-trials\", \"research\", \"data-lake\", \"trash-can\" ],\n",
    "    \"DefaultZones\": [ \"sdlc\" , \"quarantine\"]\n",
    "}\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-catalog', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-consumer', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-owner', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'community-profile', {\"KarmaPointPlateau\":\"500\"})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'glossary-view', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-science', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'subject-area', {})\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'asset-manager', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'governance-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'data-manager', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'discovery-engine', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'it-infrastructure', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'project-management', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'software-developer', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'devops', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'digital-architecture', accessServiceOptions)\n",
    "configureAccessService(adminPlatformURL, adminUserId, mdrServerName, 'design-model', accessServiceOptions)\n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Configuring the exchangeDL01 Integration Daemon\n",
    "\n",
    "The **exchangeDL01** integration daemon server supports the automatic exchange of metadata with third party technologies.\n",
    "It runs [integration connectors](https://egeria-project.org/concepts/integration-connector)\n",
    "that each connect to a particular third party technology to exchange metadata.\n",
    "\n",
    "Egeria offers the following Open Metadata Integration Services (OMIS), or integration services for short.  These integration services provide specialist services for an integration connector.  The command below lists the different types of integration services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "getIntegrationServices(exchangeDL01PlatformName, exchangeDL01PlatformURL)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An integration connector depends on a single integration service.\n",
    "Gary plans to use two integration connectors supplied by Egeria:\n",
    "\n",
    "The **DataFilesMonitorIntegrationConnector** maintains a DataFile asset for each file in the directory (or any subdirectory).\n",
    "When a new file is created, a new DataFile asset is created.  If a file is modified, the lastModified property\n",
    "of the corresponding DataFile asset is updated.  When a file is deleted, its corresponding DataFile asset is also deleted.\n",
    "\n",
    "The **DataFolderMonitorIntegrationConnector** maintains a DataFolder asset for the directory.  The files and directories\n",
    "underneath it are assumed to be elements/records in the DataFolder asset and so each time there is a change to the\n",
    "files and directories under the monitored directory, it results in an update to the lastModified property\n",
    "of the corresponding DataFolder asset.\n",
    "\n",
    "They will be used to automatically catalog data files provided by the different partner hospitals and move them from the\n",
    "landing area to the data lake once the cataloguing is complete.\n",
    "\n",
    "Figure 3 shows the integration daemon with its two connectors.\n",
    "It uses cocoMDS1 to store and retrieve metadata, since that is where the assets for the data lake are catalogued.\n",
    "\n",
    "![Figure 3](images/integration-daemon.png)\n",
    "> **Figure 3:** exchangeDL01 with its partner metadata server\n",
    "\n",
    "\n",
    "### Configuring the server\n",
    "\n",
    "The commands below configure the integration daemon with the Files Integrator OMIS and the two connectors.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daemonServerName        = \"exchangeDL01\"\n",
    "daemonServerPlatform    = dataLakePlatformURL\n",
    "daemonServerUserId      = \"exchangeDL01npa\"\n",
    "daemonServerPassword    = \"exchangeDL01passw0rd\"\n",
    "\n",
    "mdrServerName           = \"cocoMDS1\"\n",
    "mdrServerPlatform       = dataLakePlatformURL\n",
    "\n",
    "OakDeneConnectorName         = \"OakDeneLandingAreaFilesMonitor\"\n",
    "OakDeneConnectorUserId       = \"onboardDL01npa\"\n",
    "OakDeneConnectorSourceName   = \"HospitalLandingArea\"\n",
    "OakDeneConnectorFolder       = fileSystemRoot + '/landing-area/hospitals/oak-dene/clinical-trials/drop-foot'\n",
    "OakDeneConnectorConnection   =  { \n",
    "                                      \"class\" : \"Connection\",\n",
    "                                      \"connectorType\" : \n",
    "                                      {\n",
    "                                           \"class\" : \"ConnectorType\",\n",
    "                                           \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.basicfiles.DataFilesMonitorIntegrationProvider\"           \n",
    "                                      },\n",
    "                                      \"endpoint\" :\n",
    "                                      {\n",
    "                                           \"class\" : \"Endpoint\",\n",
    "                                           \"address\" : OakDeneConnectorFolder\n",
    "                                      }\n",
    "                                }\n",
    "\n",
    "OldMarketConnectorName         = \"OldMarketLandingAreaFilesMonitor\"\n",
    "OldMarketConnectorUserId       = \"onboardDL01npa\"\n",
    "OldMarketConnectorSourceName   = \"HospitalLandingArea\"\n",
    "OldMarketConnectorFolder       = fileSystemRoot + '/landing-area/hospitals/old-market/clinical-trials/drop-foot'\n",
    "OldMarketConnectorConnection   =  { \n",
    "                                      \"class\" : \"Connection\",\n",
    "                                      \"connectorType\" : \n",
    "                                      {\n",
    "                                           \"class\" : \"ConnectorType\",\n",
    "                                           \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.basicfiles.DataFilesMonitorIntegrationProvider\"           \n",
    "                                      },\n",
    "                                      \"endpoint\" :\n",
    "                                      {\n",
    "                                           \"class\" : \"Endpoint\",\n",
    "                                           \"address\" : OldMarketConnectorFolder\n",
    "                                      }\n",
    "                                 }\n",
    "\n",
    "folderConnectorName       = \"DropFootClinicalTrialResultsFolderMonitor\"\n",
    "folderConnectorUserId     = \"monitorDL01npa\"\n",
    "folderConnectorSourceName = \"DropFootClinicalTrialResults\"\n",
    "folderConnectorFolder     = fileSystemRoot + '/data-lake/research/clinical-trials/drop-foot/weekly-measurements'\n",
    "folderConnectorConnection =  { \n",
    "                                  \"class\" : \"Connection\",\n",
    "                                  \"connectorType\" : \n",
    "                                  {\n",
    "                                       \"class\" : \"ConnectorType\",\n",
    "                                       \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.basicfiles.DataFolderMonitorIntegrationProvider\"           \n",
    "                                  },\n",
    "                                  \"endpoint\" :\n",
    "                                  {\n",
    "                                       \"class\" : \"Endpoint\",\n",
    "                                       \"address\" : folderConnectorFolder\n",
    "                                  }\n",
    "                              }\n",
    "\n",
    "print(\"Configuring \" + daemonServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, daemonServerName, daemonServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, daemonServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, daemonServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, daemonServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, daemonServerName, daemonServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, daemonServerName, daemonServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, daemonServerName, serverSecurityConnectionBody)\n",
    "configureDefaultAuditLog(adminPlatformURL, adminUserId, daemonServerName)\n",
    "\n",
    "print(\"\\nConfiguring \" + daemonServerName + \" integration connectors ...\")\n",
    "\n",
    "connectorConfigs = [ \n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : OakDeneConnectorName,\n",
    "        \"connectorUserId\"             : OakDeneConnectorUserId,\n",
    "        \"connection\"                  : OakDeneConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : OakDeneConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "    },\n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : OldMarketConnectorName,\n",
    "        \"connectorUserId\"             : OldMarketConnectorUserId,\n",
    "        \"connection\"                  : OldMarketConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : OldMarketConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "    },\n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : folderConnectorName,\n",
    "        \"connectorUserId\"             : folderConnectorUserId,\n",
    "        \"connection\"                  : folderConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : folderConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "  }]\n",
    "\n",
    "configureIntegrationService(adminPlatformURL, adminUserId, daemonServerName, mdrServerName, mdrServerPlatform, \"files-integrator\", {}, connectorConfigs)\n",
    "\n",
    "print (\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Configuring the monitorDev01 Integration Daemon\n",
    "\n",
    "The **monitorDev01** is another integration daemon server supporting the development team.  It is cataloguing assets in preparation for deployment into production.  It has no integration services active initially since they are set up in the labs and demos that use it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daemonServerName        = \"monitorDev01\"\n",
    "daemonServerPlatform    = devPlatformURL\n",
    "daemonServerUserId      = \"erinoverview\"\n",
    "daemonServerPassword    = \"erinoverviewpassw0rd\"\n",
    "\n",
    "mdrServerName           = \"cocoMDSx\"\n",
    "mdrServerPlatform       = devPlatformURL\n",
    "\n",
    "print(\"Configuring \" + daemonServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, daemonServerName, daemonServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, daemonServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, daemonServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, daemonServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, daemonServerName, daemonServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, daemonServerName, daemonServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, daemonServerName, serverSecurityConnectionBody)\n",
    "configureDefaultAuditLog(adminPlatformURL, adminUserId, daemonServerName)\n",
    "\n",
    "print (\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Configuring the monitorGov01 Integration Daemon\n",
    "\n",
    "The **monitorGov01** is another integration daemon server supporting the governance team.  It is managing the capture of lineage for the governance actions and external processing in the data lake.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daemonServerName        = \"monitorGov01\"\n",
    "daemonServerPlatform    = dataLakePlatformURL\n",
    "daemonServerUserId      = \"exchangeDL01npa\"\n",
    "daemonServerPassword    = \"exchangeDL01passw0rd\"\n",
    "\n",
    "mdrServerName           = \"cocoMDS1\"\n",
    "mdrServerPlatform       = dataLakePlatformURL\n",
    "\n",
    "KafkaReceiverConnectorName         = \"KafkaOpenLineageEventReceiver\"\n",
    "KafkaReceiverConnectorUserId       = \"onboardDL01npa\"\n",
    "KafkaReceiverConnectorSourceName   = \"Apache Kafka\"\n",
    "KafkaReceiverConnectorConnection   = { \n",
    "                                      \"class\" : \"VirtualConnection\",\n",
    "                                      \"connectorType\" : \n",
    "                                      {\n",
    "                                           \"class\" : \"ConnectorType\",\n",
    "                                           \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageEventReceiverIntegrationProvider\"           \n",
    "                                      },\n",
    "                                      \"embeddedConnections\" : \n",
    "                                      [\n",
    "                                          {\n",
    "                                              \"class\": \"EmbeddedConnection\",\n",
    "                                              \"embeddedConnection\" :\n",
    "                                              {\n",
    "                                                 \"class\": \"Connection\",\n",
    "                                                 \"connectorType\": \n",
    "                                                  {\n",
    "                                                      \"class\": \"ConnectorType\",\n",
    "                                                      \"connectorProviderClassName\": \"org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider\",\n",
    "                                                  },\n",
    "                                                  \"endpoint\": \n",
    "                                                  {\n",
    "                                                      \"class\": \"Endpoint\",\n",
    "                                                      \"address\": \"openlineage.topic\"\n",
    "                                                  },\n",
    "                                                  \"configurationProperties\":\n",
    "                                                  {\n",
    "                                                      \"producer\": \n",
    "                                                      {\n",
    "                                                          \"bootstrap.servers\": \"localhost:9092\"\n",
    "                                                      },\n",
    "                                                      \"local.server.id\": \"f234e808-2d0c-4d88-83df-275eee20c1b7\",\n",
    "                                                      \"consumer\": \n",
    "                                                      {\n",
    "                                                          \"bootstrap.servers\": \"localhost:9092\"\n",
    "                                                      }\n",
    "                                                  }\n",
    "                                              }\n",
    "                                          }\n",
    "                                      ]\n",
    "                                     }\n",
    "\n",
    "GovernanceActionConnectorName         = \"GovernanceActionOpenLineageCreator\"\n",
    "GovernanceActionConnectorUserId       = \"onboardDL01npa\"\n",
    "GovernanceActionConnectorSourceName   = \"Egeria\"\n",
    "GovernanceActionConnectorConnection   = { \n",
    "                                          \"class\" : \"Connection\",\n",
    "                                          \"connectorType\" : \n",
    "                                          {\n",
    "                                             \"class\" : \"ConnectorType\",\n",
    "                                             \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.openlineage.GovernanceActionOpenLineageIntegrationProvider\"           \n",
    "                                          },\n",
    "                                        }\n",
    "\n",
    "APILoggerConnectorName         = \"APIBasedOpenLineageLogStore\"\n",
    "APILoggerConnectorUserId       = \"onboardDL01npa\"\n",
    "APILoggerConnectorSourceName   = \"Egeria\"\n",
    "APILoggerConnectorConnection   = { \n",
    "                                      \"class\" : \"Connection\",\n",
    "                                      \"connectorType\" : \n",
    "                                      {\n",
    "                                           \"class\" : \"ConnectorType\",\n",
    "                                           \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.openlineage.APIBasedOpenLineageLogStoreProvider\"           \n",
    "                                      },\n",
    "                                      \"endpoint\" :\n",
    "                                      {\n",
    "                                           \"class\" : \"Endpoint\",\n",
    "                                           \"address\" : \"http://localhost:5000/api/v1/lineage\"\n",
    "                                      }\n",
    "                                 }\n",
    "\n",
    "FileLoggerConnectorName         = \"FileBasedOpenLineageLogStore\"\n",
    "FileLoggerConnectorUserId       = \"onboardDL01npa\"\n",
    "FileLoggerConnectorSourceName   = \"Egeria\"\n",
    "FileLoggerConnectorConnection   = { \n",
    "                                      \"class\" : \"Connection\",\n",
    "                                      \"connectorType\" : \n",
    "                                      {\n",
    "                                           \"class\" : \"ConnectorType\",\n",
    "                                           \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.openlineage.FileBasedOpenLineageLogStoreProvider\"           \n",
    "                                      },\n",
    "                                      \"endpoint\" :\n",
    "                                      {\n",
    "                                           \"class\" : \"Endpoint\",\n",
    "                                           \"address\" : fileSystemRoot + '/openlineage.log'\n",
    "                                      }\n",
    "                                 }\n",
    "\n",
    "CataloguerConnectorName         = \"OpenLineageCataloguer\"\n",
    "CataloguerConnectorUserId       = \"onboardDL01npa\"\n",
    "CataloguerConnectorSourceName   = \"OpenLineageSources\"\n",
    "CataloguerConnectorConnection   = { \n",
    "                                      \"class\" : \"Connection\",\n",
    "                                      \"connectorType\" : \n",
    "                                      {\n",
    "                                           \"class\" : \"ConnectorType\",\n",
    "                                           \"connectorProviderClassName\" : \"org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageCataloguerIntegrationProvider\"           \n",
    "                                      }\n",
    "                                 }\n",
    "\n",
    "\n",
    "print(\"Configuring \" + daemonServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, daemonServerName, daemonServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, daemonServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, daemonServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, daemonServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, daemonServerName, daemonServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, daemonServerName, daemonServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, daemonServerName, serverSecurityConnectionBody)\n",
    "configureDefaultAuditLog(adminPlatformURL, adminUserId, daemonServerName)\n",
    "\n",
    "print(\"\\nConfiguring \" + daemonServerName + \" integration connectors ...\")\n",
    "\n",
    "connectorConfigs = [ \n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : KafkaReceiverConnectorName,\n",
    "        \"connectorUserId\"             : KafkaReceiverConnectorUserId,\n",
    "        \"connection\"                  : KafkaReceiverConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : KafkaReceiverConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "    },\n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : GovernanceActionConnectorName,\n",
    "        \"connectorUserId\"             : GovernanceActionConnectorUserId,\n",
    "        \"connection\"                  : GovernanceActionConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : GovernanceActionConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "    },\n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : APILoggerConnectorName,\n",
    "        \"connectorUserId\"             : APILoggerConnectorUserId,\n",
    "        \"connection\"                  : APILoggerConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : APILoggerConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "    },\n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : FileLoggerConnectorName,\n",
    "        \"connectorUserId\"             : FileLoggerConnectorUserId,\n",
    "        \"connection\"                  : FileLoggerConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : FileLoggerConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "    },\n",
    "    {\n",
    "        \"class\"                       : \"IntegrationConnectorConfig\",\n",
    "        \"connectorName\"               : CataloguerConnectorName,\n",
    "        \"connectorUserId\"             : CataloguerConnectorUserId,\n",
    "        \"connection\"                  : CataloguerConnectorConnection,\n",
    "        \"metadataSourceQualifiedName\" : CataloguerConnectorSourceName,\n",
    "        \"refreshTimeInterval\"         : 10,\n",
    "        \"usesBlockingCalls\"           : \"false\"\n",
    "  }]\n",
    "\n",
    "configureIntegrationService(adminPlatformURL, adminUserId, daemonServerName, mdrServerName, mdrServerPlatform, \"lineage-integrator\", {}, connectorConfigs)\n",
    "\n",
    "print (\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Configuring governDL01 Governance Engine Hosting Server\n",
    "\n",
    "The Engine Host OMAG server is a special kind of governance server that hosts one or more governance engines.\n",
    "\n",
    "A governance engine is a set of specialized services that perform specific functions to manage the digital landscape and the metadata that describes it.  \n",
    "\n",
    "### Automated metadata discovery\n",
    "\n",
    "One example of a type of governance engine is a discovery engine. The discovery engine runs discovery services.  Discovery services analyze the content of a real-world artifact or resource.  For example, a discovery service may open up a data set and assess the quality of the data inside.\n",
    "\n",
    "The result of a discovery service's analysis is stored in a metadata server as a discovery analysis report that is chained off of the asset's definition.  This report can be retrieved either through the engine host server's API or through the metadata server's APIs, specifically the Discovery Engine OMAS and the Asset Owner OMAS.\n",
    "\n",
    "The interfaces used by discovery services are defined in the [Open Discovery Framework (ODF)](https://egeria-project.org/frameworks/odf/overview/).  This framework enables new implementations of discovery services to be deployed to the discovery engines.\n",
    "\n",
    "\n",
    "\n",
    "### Automated governance\n",
    "\n",
    "Another type of governance engine is a governance action engine.  The governance action engine runs governance action services.\n",
    "Governance action services monitor the asset metadata and verify that it is set up correctly, determine how to fix anomalies, errors and omissions,\n",
    "make the necessary changes and provision real-world artifacts and resources based on the resulting metadata.\n",
    "\n",
    "\n",
    "\n",
    "### Understanding the engine services\n",
    "\n",
    "Coco Pharmaceuticals runs one engine host server for its data lake.  It is called `governDL01` and it runs on the data lake platform.\n",
    "Within the engine host server there are engine services.  Each engine service supports a specific type of governance engine.\n",
    "The command below shows you the different types of engine services\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "getEngineServices(governDL01PlatformName, governDL01PlatformURL)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "The governDL01 server is the Engine Host server that runs governance functions that monitor, validate, correct and enrich metadata for use by all of the technologies in the connected open metadata ecosystem.\n",
    "\n",
    "The **Asset Analysis** Open Metadata Engine Service (OMES) is responsible for running discovery engines from the\n",
    "[Open Discovery Framework (ODF)](https://egeria-project.org/frameworks/odf/overview/).\n",
    "Coco Pharmaceuticals has two discovery engines:\n",
    "\n",
    "* **AssetDiscovery** - extracts metadata about different types of assets on request.\n",
    "* **AssetQuality** - assesses the quality of the content of assets on request.\n",
    "\n",
    "The **Governance Action** Open Metadata Engine Service (OMES) is responsible for running governance action engines from the\n",
    "[Governance Action Framework (GAF)](https://egeria-project.org/frameworks/gaf/overview/).\n",
    "Coco Pharmaceuticals has one governance action engine:\n",
    "\n",
    "* **AssetGovernance** - monitors for new assets in the landing areas, automatically curates them and provisions them in the data lake.\n",
    "\n",
    "Figure 4 shows the integration daemon with its two connectors.\n",
    "It uses cocoMDS1 to store and retrieve metadata, since that is where the assets for the data lake are catalogued.\n",
    "\n",
    "![Figure 4](images/engine-host.png)\n",
    "> **Figure 4:** Metadata servers for governDL01\n",
    "\n",
    "### Configuring the server\n",
    "\n",
    "The commands below configure the engine host server with the Asset Analysis OMES and Governance Action OMES.  \n",
    "The definitions of the named governance engines and their services are retrieved from the `cocoMDS1` metadata server through its Governance Engine OMAS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "engineServerName        = \"governDL01\"\n",
    "engineServerPlatform    = dataLakePlatformURL\n",
    "engineServerUserId      = \"governDL01npa\"\n",
    "engineServerPassword    = \"governDL01passw0rd\"\n",
    "\n",
    "engineServerMDRName     = \"cocoMDS2\"\n",
    "engingServerMDRPlatform = corePlatformURL\n",
    "\n",
    "mdrServerName           = \"cocoMDS1\"\n",
    "mdrServerPlatform       = dataLakePlatformURL\n",
    "\n",
    "print(\"Configuring \" + engineServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, engineServerName, engineServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, engineServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, engineServerName)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, engineServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, engineServerName, engineServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, engineServerName, engineServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, engineServerName, serverSecurityConnectionBody)\n",
    "configureDefaultAuditLog(adminPlatformURL, adminUserId, engineServerName)\n",
    "\n",
    "print(\"\\nConfiguring \" + engineServerName + \" engines ...\")\n",
    "\n",
    "configureEngineDefinitionServices(adminPlatformURL, adminUserId, engineServerName, engineServerMDRName, engingServerMDRPlatform)\n",
    "\n",
    "discoveryEngines = [ \n",
    "    {\n",
    "        \"class\"               : \"EngineConfig\",\n",
    "        \"engineQualifiedName\" : \"AssetDiscovery\",\n",
    "        \"engineUserId\"        : \"findItDL01npa\"\n",
    "    }, \n",
    "    {\n",
    "        \"class\"               : \"EngineConfig\",\n",
    "        \"engineQualifiedName\" : \"AssetQuality\",\n",
    "        \"engineUserId\"        : \"findItDL01npa\"\n",
    "    }]\n",
    "\n",
    "governanceActionEngines = [ \n",
    "    {\n",
    "        \"class\"               : \"EngineConfig\",\n",
    "        \"engineQualifiedName\" : \"AssetGovernance\",\n",
    "        \"engineUserId\"        : \"findItDL01npa\"\n",
    "    }]\n",
    "\n",
    "configureGovernanceEngineService(adminPlatformURL, adminUserId, engineServerName, mdrServerName, mdrServerPlatform, \"asset-analysis\", discoveryEngines)\n",
    "configureGovernanceEngineService(adminPlatformURL, adminUserId, engineServerName, mdrServerName, mdrServerPlatform, \"governance-action\", governanceActionEngines)\n",
    "\n",
    "print (\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Configuring the cocoOLS1  - Open Lineage Governance Server\n",
    "\n",
    "The open lineage server **cocoOLS1** is a dedicated Governance Server that manages a historical warehouse of lineage information.\n",
    "The Open Lineage Services provides a historic reporting warehouse for lineage. It listens to events that are sent out by the Asset Lineage OMAS, and stores lineage data in a graph database. This lineage can then be queried through the Open Lineage Services Client and by its REST API, for example by a lineage GUI.\n",
    "\n",
    "![Figure 5](images/open-lineage-server.png)\n",
    "> **Figure 5:** Open Lineage Governance Server and related services"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring the server\n",
    "\n",
    "The admin commands below will configure new server with name `cocoOLS1` on the Data Lake Platform.\n",
    "\n",
    "The server configuration document includes a `OpenLineageConfig` section. Complete instructions can be found on [Open Lineage Server configuration](https://egeria-project.org/guides/admin/servers/configuring-an-open-lineage-server/) page.\n",
    "\n",
    "For this lab we use following:\n",
    "\n",
    "- `accessServiceConfig`- Lineage events are received from the output topic connector of Asset Lineage OMAS. Also in some cases when needed, Open Lineage Server will use Asset Lineage OMAS REST client to fetch metadata needed to build detailed graph context. \n",
    "- `lineageGraphConnection` - Default configuration for the lineage graph store connector. The default graph store implementation is using `JanusGraph` database, `BerkeleyDB` as local storage and `Lucene` as search index. For more details about the connector see [Lineage Janus Connector](https://egeria-project.org/guides/admin/servers/configuring-an-open-lineage-server/)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lineageServerName        = \"cocoOLS1\"\n",
    "lineageServerPlatform    = dataLakePlatformURL\n",
    "\n",
    "mdrServerName            = \"cocoMDS1\"\n",
    "mdrServerUserId          = \"cocoMDS1npa\"\n",
    "mdrServerPassword        = \"secret\"\n",
    "mdrServerPlatform        = dataLakePlatformURL\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OLS Server configuration common function\n",
    "#\n",
    "#  adminPlatformURL - url of the platform where admin-services are running\n",
    "#  adminUserId - user with admin access\n",
    "#  lineageServerName - name of the new server initializing Open Lineage Services\n",
    "#  metadataPlatformURL - url of the platform hosting the metadata repository server\n",
    "#  metadataServerName - name of the server where metadata repository is located and asset-lineage omas is running\n",
    "#  metadataServerUserName - user name with access asset-lineage omas\n",
    "#  metadataServerUserPassword - password for the user with access to asset-lineage omas\n",
    "\n",
    "\n",
    "def configureOpenLineageService(adminPlatformURL, adminUserId, lineageServerName, metadataPlatformURL, metadataServerName, metadataServerUserName, metadataServerUserPassword):\n",
    "    adminCommandURLRoot = adminPlatformURL + '/open-metadata/admin-services/users/' + adminUserId + '/servers/'\n",
    "    print (\"   ... configuring the Open Lineage Services server \" + lineageServerName + \" ...\")\n",
    "    jsonContentHeader = {'content-type':'application/json'}\n",
    "    requestBody =  {\n",
    "       \"class\": \"OpenLineageConfig\",\n",
    "       \"openLineageDescription\": \"Open Lineage Service is used for the storage and querying of lineage\",\n",
    "       \"lineageGraphConnection\": {\n",
    "         \"class\": \"Connection\",\n",
    "         \"displayName\": \"Lineage Graph Connection\",\n",
    "         \"description\": \"Used for storing lineage in the Open Metadata format\",\n",
    "         \"connectorType\": {\n",
    "           \"class\": \"ConnectorType\",\n",
    "           \"connectorProviderClassName\": \"org.odpi.openmetadata.openconnectors.governancedaemonconnectors.openlineageconnectors.janusconnector.graph.LineageGraphConnectorProvider\"\n",
    "         },\n",
    "         \"configurationProperties\": {\n",
    "           \"gremlin.graph\": \"org.janusgraph.core.JanusGraphFactory\",\n",
    "           \"storage.backend\": \"berkeleyje\",\n",
    "           \"storage.directory\": \"data/servers/\"+ lineageServerName + \"/lineage-repository/berkeley\",\n",
    "           \"index.search.backend\": \"lucene\",\n",
    "           \"index.search.directory\": \"data/servers/\"+ lineageServerName + \"/lineage-repository/searchindex\"\n",
    "         }\n",
    "       },\n",
    "       \"accessServiceConfig\": {\n",
    "         \"serverName\": metadataServerName,\n",
    "         \"serverPlatformUrlRoot\": metadataPlatformURL,\n",
    "         \"user\": metadataServerUserName,\n",
    "         \"password\": metadataServerUserPassword\n",
    "       },\n",
    "       \"backgroundJobs\": [\n",
    "         {\n",
    "           \"jobName\": \"LineageGraphJob\",\n",
    "           \"jobInterval\": 120,\n",
    "           \"jobEnabled\": \"false\"\n",
    "         }, \n",
    "         {\n",
    "           \"jobName\": \"AssetLineageUpdateJob\",\n",
    "           \"jobInterval\": 120,\n",
    "           \"jobEnabled\": \"false\",\n",
    "           \"jobDefaultValue\": \"2021-12-03T10:15:30\"\n",
    "         }\n",
    "       ]\n",
    "    }\n",
    "    print(requestBody)\n",
    "    url = adminCommandURLRoot + lineageServerName + '/open-lineage/configuration'\n",
    "    postAndPrintResult(url, json=requestBody, headers=jsonContentHeader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This server type also depends on standard sub-systems like event bus and audit logging. \n",
    "\n",
    "To configure follow the command sequence below. Once it is finished successfully it will print out 'Done.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Configuring \" + lineageServerName + \" Open Lineage Services (OLS) ...\")\n",
    "\n",
    "configureEventBus(adminPlatformURL, adminUserId, lineageServerName, eventBusBody)\n",
    "configureDefaultAuditLog(adminPlatformURL, adminUserId, lineageServerName)\n",
    "configureOpenLineageService(adminPlatformURL, adminUserId, lineageServerName, mdrServerPlatform, mdrServerName, mdrServerUserId, mdrServerPassword)\n",
    "\n",
    "print (\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configuring the View Server and View Services"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Egeria's UI allows Coco Pharmaceutical's employees to understand more\n",
    "about their metadata environment.\n",
    "This UI uses special services, called view services, that run in an Egeria View Server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "getViewServices(cocoView1PlatformName, cocoView1PlatformURL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an initial version of an example to configure the view services.\n",
    "Since this area is still in development the configuration is likely to change, and so all of the\n",
    "functions are in this section of the notebook rather than consolidated with our common functions.\n",
    "\n",
    "The new UI is deployed in the k8s environment\n",
    "\n",
    "The tenant (`coco` in this case) must be explicitly provided in the URL, as must navigation to the login page\n",
    "For example if the UI is on port 18091,login at https://localhost:18091/coco/login\n",
    "\n",
    "Further docs will be added in future releases. Please use http://slack.lfai.foundation to get further help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def configureGovernanceSolutionViewService(adminPlatformURL, adminUserId, viewServerName, viewService, remotePlatformURL,remoteServerName):\n",
    "    adminCommandURLRoot = adminPlatformURL + '/open-metadata/admin-services/users/' + adminUserId + '/servers/'\n",
    "    print (\"   ... configuring the \" + viewService + \" Governance Solution View Service for this server...\")\n",
    "    url = adminCommandURLRoot + viewServerName + '/view-services/' + viewService\n",
    "    jsonContentHeader = {'content-type':'application/json'}\n",
    "    viewBody = {\n",
    "        \"class\": \"ViewServiceConfig\",\n",
    "        \"omagserverPlatformRootURL\": remotePlatformURL,\n",
    "        \"omagserverName\" : remoteServerName\n",
    "    }\n",
    "    postAndPrintResult(url, json=viewBody, headers=jsonContentHeader)\n",
    "    \n",
    "def configureIntegrationViewService(adminPlatformURL, adminUserId, viewServerName, viewService, configBody):\n",
    "    adminCommandURLRoot = adminPlatformURL + '/open-metadata/admin-services/users/' + adminUserId + '/servers/'\n",
    "    print (\"   ... configuring the \" + viewService + \" Integration View Service for this server...\")\n",
    "    url = adminCommandURLRoot + viewServerName + '/view-services/' + viewService\n",
    "    jsonContentHeader = {'content-type':'application/json'}\n",
    "    postAndPrintResult(url, json=configBody, headers=jsonContentHeader)\n",
    "\n",
    "\n",
    "# A view server supports the presentation server UI (a node based app). Here we run it on the datalake platform    \n",
    "viewServerName          = \"cocoView1\"\n",
    "viewServerUserId        = \"cocoView1npa\"\n",
    "viewServerPassword      = \"cocoView1passw0rd\"\n",
    "viewServerPlatform      = dataLakePlatformURL\n",
    "viewServerType          = \"View Server\"\n",
    "\n",
    "# Configuration is similar to most servers\n",
    "print(\"Configuring \" + viewServerName + \"...\")\n",
    "\n",
    "configurePlatformURL(adminPlatformURL, adminUserId, viewServerName, viewServerPlatform)\n",
    "configureMaxPageSize(adminPlatformURL, adminUserId, mdrServerName, maxPageSize)\n",
    "clearServerType(adminPlatformURL, adminUserId, viewServerName)\n",
    "configureServerType(adminPlatformURL,adminUserId,viewServerName,viewServerType)\n",
    "configureOwningOrganization(adminPlatformURL, adminUserId, viewServerName, organizationName)\n",
    "configureUserId(adminPlatformURL, adminUserId, viewServerName, viewServerUserId)\n",
    "configurePassword(adminPlatformURL, adminUserId, viewServerName, viewServerPassword)\n",
    "configureSecurityConnection(adminPlatformURL, adminUserId, viewServerName, serverSecurityConnectionBody)\n",
    "configureEventBus(adminPlatformURL, adminUserId, viewServerName, eventBusBody)\n",
    "configureDefaultAuditLog(adminPlatformURL, adminUserId, viewServerName)\n",
    "\n",
    "# The governance solution view services currently only consist of glossary author\n",
    "print (\"Configuring the Governance Solution View Services\")\n",
    "remotePlatformURL=corePlatformURL\n",
    "remoteServerName=\"cocoMDS2\"\n",
    "viewService=\"glossary-author\"\n",
    "configureGovernanceSolutionViewService(adminPlatformURL, adminUserId, viewServerName, viewService, remotePlatformURL,remoteServerName)\n",
    "\n",
    "print (\"Configuring the Integration View Services\")\n",
    "\n",
    "# repository explorer integration view service\n",
    "viewService=\"rex\"\n",
    "rexConfigBody = {\n",
    "            \"class\":\"IntegrationViewServiceConfig\",\n",
    "            \"viewServiceAdminClass\":\"org.odpi.openmetadata.viewservices.rex.admin.RexViewAdmin\",\n",
    "            \"viewServiceFullName\":\"Repository Explorer\",\n",
    "            \"viewServiceOperationalStatus\":\"ENABLED\",\n",
    "            \"omagserverPlatformRootURL\": \"UNUSED\",\n",
    "            \"omagserverName\" : \"UNUSED\",\n",
    "            \"resourceEndpoints\" : [\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Core Platform\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"platformRootURL\"    : corePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"DataLake Platform\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"platformRootURL\"    : dataLakePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Development Platform\",\n",
    "                    \"platformName\"       : \"Development\",\n",
    "                    \"platformRootURL\"    : devPlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS1\",\n",
    "                    \"description\"        : \"Data Lake Operations\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"serverName\"         : \"cocoMDS1\"\n",
    "                },\n",
    "                 {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS2\",\n",
    "                    \"description\"        : \"Governance\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS2\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS3\",\n",
    "                    \"description\"        : \"Research\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS3\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS5\",\n",
    "                    \"description\"        : \"Business Systems\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS5\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS6\",\n",
    "                    \"description\"        : \"Manufacturing\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS6\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDSx\",\n",
    "                    \"description\"        : \"Development\",\n",
    "                    \"platformName\"       : \"Development\",\n",
    "                    \"serverName\"         : \"cocoMDSx\"\n",
    "                },\n",
    "            ]\n",
    "        }\n",
    "\n",
    "configureIntegrationViewService(adminPlatformURL, adminUserId, viewServerName, viewService, rexConfigBody)\n",
    "\n",
    "# type-explorer has endpoints\n",
    "viewService=\"tex\"\n",
    "texConfigBody = {\n",
    "            \"class\":\"IntegrationViewServiceConfig\",\n",
    "            \"viewServiceAdminClass\":\"org.odpi.openmetadata.viewservices.tex.admin.TexViewAdmin\",\n",
    "            \"viewServiceFullName\":\"Type Explorer\",\n",
    "            \"viewServiceOperationalStatus\":\"ENABLED\",\n",
    "            \"omagserverPlatformRootURL\": \"UNUSED\",\n",
    "            \"omagserverName\" : \"UNUSED\",\n",
    "            \"resourceEndpoints\" : [\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Core Platform\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"platformRootURL\"    : corePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"DataLake Platform\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"platformRootURL\"    : dataLakePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Development Platform\",\n",
    "                    \"platformName\"       : \"Development\",\n",
    "                    \"platformRootURL\"    : devPlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS1\",\n",
    "                    \"description\"        : \"Data Lake Operations\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"serverName\"         : \"cocoMDS1\"\n",
    "                },\n",
    "                 {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS2\",\n",
    "                    \"description\"        : \"Governance\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS2\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS3\",\n",
    "                    \"description\"        : \"Research\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS3\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS5\",\n",
    "                    \"description\"        : \"Business Systems\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS5\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS6\",\n",
    "                    \"description\"        : \"Manufacturing\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS6\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDSx\",\n",
    "                    \"description\"        : \"Development\",\n",
    "                    \"platformName\"       : \"Development\",\n",
    "                    \"serverName\"         : \"cocoMDSx\"\n",
    "                },\n",
    "            ]\n",
    "        }\n",
    "\n",
    "configureIntegrationViewService(adminPlatformURL, adminUserId, viewServerName, viewService, texConfigBody)\n",
    "\n",
    "# Dino provides insight into the operational environment of egeria - this config body allows coco's platforms & servers to be accessed\n",
    "viewService=\"dino\"\n",
    "DinoConfigBody = {\n",
    "            \"class\":\"IntegrationViewServiceConfig\",\n",
    "            \"viewServiceAdminClass\":\"org.odpi.openmetadata.viewservices.dino.admin.DinoViewAdmin\",\n",
    "            \"viewServiceFullName\":\"Dino\",\n",
    "            \"viewServiceOperationalStatus\":\"ENABLED\",\n",
    "            \"omagserverPlatformRootURL\": \"UNUSED\",\n",
    "            \"omagserverName\" : \"UNUSED\",\n",
    "            \"resourceEndpoints\" : [\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Core Platform\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"platformRootURL\"    : corePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"DataLake Platform\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"platformRootURL\"    : dataLakePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Development Platform\",\n",
    "                    \"platformName\"       : \"Development\",\n",
    "                    \"platformRootURL\"    : devPlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS1\",\n",
    "                    \"description\"        : \"Data Lake Operations\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"serverName\"         : \"cocoMDS1\"\n",
    "                },\n",
    "                 {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS2\",\n",
    "                    \"description\"        : \"Governance\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS2\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS3\",\n",
    "                    \"description\"        : \"Research\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS3\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS4\",\n",
    "                    \"description\"        : \"Data Lake Users\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"serverName\"         : \"cocoMDS4\"\n",
    "                },\n",
    "                              {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS5\",\n",
    "                    \"description\"        : \"Business Systems\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS5\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDS6\",\n",
    "                    \"description\"        : \"Manufacturing\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"serverName\"         : \"cocoMDS6\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoMDSx\",\n",
    "                    \"description\"        : \"Development\",\n",
    "                    \"platformName\"       : \"Development\",\n",
    "                    \"serverName\"         : \"cocoMDSx\"\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Server\",\n",
    "                    \"serverInstanceName\" : \"cocoView1\",\n",
    "                    \"description\"        : \"View Server\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"serverName\"         : \"cocoView1\"\n",
    "                },\n",
    "            ]\n",
    "        }\n",
    "\n",
    "configureIntegrationViewService(adminPlatformURL, adminUserId, viewServerName, viewService, DinoConfigBody)\n",
    "\n",
    "# server-author endpoints\n",
    "viewService=\"server-author\"\n",
    "serverAuthorConfigBody = {\n",
    "            \"class\":\"IntegrationViewServiceConfig\",\n",
    "            \"viewServiceAdminClass\":\"org.odpi.openmetadata.viewservices.serverauthor.admin.ServerAuthorViewAdmin\",\n",
    "            \"viewFullServiceName\":\"ServerAuthor\",\n",
    "            \"viewServiceOperationalStatus\":\"ENABLED\",\n",
    "            \"viewServiceFullName\":\"Type Explorer\",\n",
    "            \"viewServiceOperationalStatus\":\"ENABLED\",\n",
    "            \"omagserverPlatformRootURL\":dataLakePlatformURL,\n",
    "            \"omagserverName\" : \"UNUSED\",\n",
    "            \"resourceEndpoints\" : [\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Core Platform\",\n",
    "                    \"platformName\"       : \"Core\",\n",
    "                    \"platformRootURL\"    : corePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"DataLake Platform\",\n",
    "                    \"platformName\"       : \"DataLake\",\n",
    "                    \"platformRootURL\"    : dataLakePlatformURL\n",
    "                },\n",
    "                {\n",
    "                    \"class\"              : \"ResourceEndpointConfig\",\n",
    "                    \"resourceCategory\"   : \"Platform\",\n",
    "                    \"description\"        : \"Development Platform\",\n",
    "                    \"platformName\"       : \"Development\",\n",
    "                    \"platformRootURL\"    : devPlatformURL\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "\n",
    "configureIntegrationViewService(adminPlatformURL, adminUserId, viewServerName, viewService, serverAuthorConfigBody)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "print (\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploying server configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The commands that have been issued so far have created a configuration document for each server.\n",
    "These configuration documents are currently local to the Development OMAG Server Platform where the\n",
    "administration commands were issued (figure 6).\n",
    "\n",
    "![Figure 6](images/creating-configuration-documents.png)\n",
    "> **Figure 6:** Creating configuration documents using administration commands\n",
    "\n",
    "If servers are to be started on the other server platforms then their configuration documents\n",
    "need to be deployed (copied) to these platforms (figure 7).\n",
    "\n",
    "![Figure 7](images/deploying-configuration-documents.png)\n",
    "> **Figure 7:** Deploying configuration documents\n",
    "\n",
    "However, before deploying the configuration documents, the receiving OMAG Server Platforms\n",
    "need to be running.\n",
    "\n",
    "The code below checks the Core and Data Lake OMAG Server Platforms are running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\nChecking OMAG Server Platform availability...\")\n",
    "\n",
    "checkServerPlatform(\"Data Lake Platform\", dataLakePlatformURL)\n",
    "checkServerPlatform(\"Core Platform\", corePlatformURL)\n",
    "checkServerPlatform(\"Dev Platform\", devPlatformURL)\n",
    "\n",
    "print (\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "Make sure the each of the platforms is running.\n",
    "\n",
    "----\n",
    "The commands below deploy the server configuration documents to the server platforms where the\n",
    "servers will run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "print(\"\\nDeploying server configuration documents to appropriate platforms...\")\n",
    "    \n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoMDS1\", dataLakePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoMDS2\", corePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoMDS3\", corePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoMDS4\", dataLakePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoMDS5\", corePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoMDS6\", corePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoMDSx\", devPlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"exchangeDL01\", dataLakePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"monitorDev01\", devPlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"governDL01\", dataLakePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoView1\", dataLakePlatformURL)\n",
    "deployServerToPlatform(adminPlatformURL, adminUserId, \"cocoOLS1\", dataLakePlatformURL)\n",
    "\n",
    "print(\"\\nDone.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}