IBM Cloud/Azure/notebooks/ML service/Credit scoring endpoint wrapper with payload logging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logging the payload with scoring endpoint wrapper\n",
    "\n",
    "Now, you're ready to create credit risk scoring endpoint wrapper. The goal of the wrapper is to automatically log all input and output payloads from credit risk deployment (created previously).\n",
    "\n",
    "**Note:** This notebook works correctly with kernel **`IBM Runtime 22.1 on Python 3.9 XS`** if using IBM Watson Studio or else use standard Python 3.9 runtime.\n",
    "\n",
    "## Prerequisites to this tutorial:\n",
    "> * Credit Risk model deployment on Azure ML Service (scoring endpoint)\n",
    "\n",
    "\n",
    "In this part of the tutorial, you use Azure Machine Learning service to:\n",
    "\n",
    "> * Set up your testing environment\n",
    "> * Retrieve the deployment from your workspace\n",
    "> * Test the deployment locally\n",
    "> * Deploy the wrapper to ACI\n",
    "> * Test the wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have not already, provision an instance of IBM Watson OpenScale using the [OpenScale link in the Cloud catalog](https://cloud.ibm.com/catalog/services/watson-openscale).\n",
    "\n",
    "Your Cloud API key can be generated by going to the [**Users** section of the Cloud console](https://cloud.ibm.com/iam#/users). From that page, click your name, scroll down to the **API Keys** section, and click **Create an IBM Cloud API key**. Give your key a name and click **Create**, then copy the created key and paste it below.\n",
    "\n",
    "**NOTE:** You can also get OpenScale `API_KEY` using IBM CLOUD CLI.\n",
    "\n",
    "How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
    "\n",
    "How to get api key using console:\n",
    "```\n",
    "bx login --sso\n",
    "bx iam api-key-create 'my_key'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLOUD_API_KEY = '***'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DB_CREDENTIALS=None\n",
    "#DB_CREDENTIALS= {\"hostname\":\"\",\"username\":\"\",\"password\":\"\",\"database\":\"\",\"port\":\"\",\"ssl\":True,\"sslmode\":\"\",\"certificate_base64\":\"\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save training data to COS for Openscale reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\"\n",
    "COS_API_KEY_ID = \"***\"\n",
    "COS_RESOURCE_CRN = \"***\" # eg \"crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003abfb5d29761c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::\"\n",
    "COS_ENDPOINT = \"***\" # Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints\n",
    "BUCKET_NAME = \"***\" #example: \"credit-risk-training-data\"\n",
    "training_data_file_name=\"credit_risk_training.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf credit_risk_training.csv\n",
    "!wget \"https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/credit_risk_training.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ibm_boto3\n",
    "from ibm_botocore.client import Config, ClientError\n",
    "\n",
    "cos_client = ibm_boto3.resource(\"s3\",\n",
    "    ibm_api_key_id=COS_API_KEY_ID,\n",
    "    ibm_service_instance_id=COS_RESOURCE_CRN,\n",
    "    ibm_auth_endpoint=\"https://iam.bluemix.net/oidc/token\",\n",
    "    config=Config(signature_version=\"oauth\"),\n",
    "    endpoint_url=COS_ENDPOINT\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(training_data_file_name, \"rb\") as file_data:\n",
    "    cos_client.Object(BUCKET_NAME, training_data_file_name).upload_fileobj(\n",
    "        Fileobj=file_data\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data_df = pd.read_csv(\"credit_risk_training.csv\",\n",
    "                    dtype={'LoanDuration': int, 'LoanAmount': int, 'InstallmentPercent': int, 'CurrentResidenceDuration': int, 'Age': int, 'ExistingCreditsCount': int, 'Dependents': int})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install `ibm-watson-openscale` package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1\n",
    "!pip install --upgrade azureml-core --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Action: Restart the kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "register model from file"
    ]
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "from azureml.core import Workspace\n",
    "from azureml.core.authentication import InteractiveLoginAuthentication\n",
    "\n",
    "az_ml_service_credentials = {'tenant_id': '***',\n",
    "                             'subscription_id': '***', \n",
    "                             'resource_group': '***',\n",
    "                             'workspace_name': '***'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_auth = InteractiveLoginAuthentication(tenant_id=az_ml_service_credentials['tenant_id'])\n",
    "\n",
    "ws = Workspace(subscription_id=az_ml_service_credentials['subscription_id'], \n",
    "               resource_group=az_ml_service_credentials['resource_group'], \n",
    "               workspace_name=az_ml_service_credentials['workspace_name'], \n",
    "               auth=interactive_auth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws.write_config()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "print('Workspace name: ' + ws.name, \n",
    "      'Azure region: ' + ws.location, \n",
    "      'Subscription id: ' + ws.subscription_id, \n",
    "      'Resource group: ' + ws.resource_group, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the environment\n",
    "\n",
    "Start by setting up a testing environment.\n",
    "\n",
    "### Import packages\n",
    "\n",
    "Import the Python packages needed for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "check version"
    ]
   },
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "import os \n",
    "\n",
    "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve deployments\n",
    "\n",
    "You created a model deployment in your workspace in the previous tutorial. Now, let's list the deployments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice\n",
    "\n",
    "credit_risk_deployment_name = '***' # service deployment name\n",
    "credit_risk_scoring_endpoint = None\n",
    "\n",
    "webservices = AciWebservice.list(ws)\n",
    "for service in webservices:\n",
    "    if service.name == credit_risk_deployment_name:\n",
    "        credit_risk_scoring_endpoint = service.scoring_uri\n",
    "        \n",
    "print('scoring endpoint', credit_risk_scoring_endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test endpoint locally\n",
    "\n",
    "Before creating a scoring endpoint wrapper let's test the original scoring endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_data = {\"input\":[{'CheckingStatus': \"0_to_200\", 'LoanDuration': 31, 'CreditHistory': \"credits_paid_to_date\", 'LoanPurpose': \"other\",'LoanAmount': 1889, 'ExistingSavings': \"100_to_500\",'EmploymentDuration': \"less_1\",'InstallmentPercent': 3,'Sex': \"female\",'OthersOnLoan': \"none\",'CurrentResidenceDuration': 3, 'OwnsProperty': \"savings_insurance\",'Age': 32,'InstallmentPlans': \"none\",'Housing': \"own\",'ExistingCreditsCount': 1,'Job': \"skilled\",'Dependents': 1,'Telephone': \"none\",'ForeignWorker': \"yes\",\n",
    "                        },\n",
    "                        {'CheckingStatus': \"no_checking\", 'LoanDuration': 13, 'CreditHistory': \"credits_paid_to_date\", 'LoanPurpose': \"car_new\",'LoanAmount': 1389, 'ExistingSavings': \"100_to_500\",'EmploymentDuration': \"1_to_4\",'InstallmentPercent': 2,'Sex': \"male\",'OthersOnLoan': \"none\",'CurrentResidenceDuration': 3, 'OwnsProperty': \"savings_insurance\",'Age': 25,'InstallmentPlans': \"none\",'Housing': \"own\",'ExistingCreditsCount': 2,'Job': \"skilled\",'Dependents': 2,'Telephone': \"none\",'ForeignWorker': \"yes\",\n",
    "                        }]\n",
    "              }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "headers = {'Content-Type':'application/json'}\n",
    "resp = requests.post(credit_risk_scoring_endpoint, json=scoring_data, headers=headers)\n",
    "output_data = json.loads(resp.json())\n",
    "\n",
    "print(\"POST to url\", credit_risk_scoring_endpoint)\n",
    "print(output_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add service provider in Openscale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n",
    "\n",
    "from ibm_watson_openscale import *\n",
    "from ibm_watson_openscale.supporting_classes.enums import *\n",
    "from ibm_watson_openscale.supporting_classes import *\n",
    "\n",
    "\n",
    "authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n",
    "#authenticator = BearerTokenAuthenticator(bearer_token=IAM_TOKEN) ## uncomment if using IAM token\n",
    "wos_client = APIClient(authenticator=authenticator)\n",
    "wos_client.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wos_client.data_marts.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_marts = wos_client.data_marts.list().result.data_marts\n",
    "if len(data_marts) == 0:\n",
    "    if DB_CREDENTIALS is not None:\n",
    "        if SCHEMA_NAME is None: \n",
    "            print(\"Please specify the SCHEMA_NAME and rerun the cell\")\n",
    "\n",
    "        print('Setting up external datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\",\n",
    "                database_configuration=DatabaseConfigurationRequest(\n",
    "                  database_type=DatabaseType.POSTGRESQL, # if using DB2 use DatabaseType.DB2\n",
    "                    credentials=PrimaryStorageCredentialsLong(\n",
    "                        hostname=DB_CREDENTIALS['hostname'],\n",
    "                        username=DB_CREDENTIALS['username'],\n",
    "                        password=DB_CREDENTIALS['password'],\n",
    "                        db=DB_CREDENTIALS['database'],\n",
    "                        port=DB_CREDENTIALS['port'],\n",
    "                        ssl=True,\n",
    "                        sslmode=DB_CREDENTIALS['sslmode'],\n",
    "                        certificate_base64=DB_CREDENTIALS['certificate_base64']\n",
    "                    ),\n",
    "                    location=LocationSchemaName(\n",
    "                        schema_name= SCHEMA_NAME\n",
    "                    )\n",
    "                )\n",
    "             ).result\n",
    "    else:\n",
    "        print('Setting up internal datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\", \n",
    "                internal_database = True).result\n",
    "        \n",
    "    data_mart_id = added_data_mart_result.metadata.id\n",
    "    \n",
    "else:\n",
    "    data_mart_id=data_marts[0].metadata.id\n",
    "    print('Using existing datamart {}'.format(data_mart_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AZURE_ENGINE_CREDENTIALS = {\n",
    "    \"client_id\": \"***\",\n",
    "    \"client_secret\": \"***\",\n",
    "    \"subscription_id\": \"***\",\n",
    "    \"tenant\": \"***\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SERVICE_PROVIDER_NAME = \"Azure Machine Learning service\"\n",
    "SERVICE_PROVIDER_DESCRIPTION = \"Added by Azure tutorial WOS notebook.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_providers = wos_client.service_providers.list().result.service_providers\n",
    "for service_provider in service_providers:\n",
    "    service_instance_name = service_provider.entity.name\n",
    "    if service_instance_name == SERVICE_PROVIDER_NAME:\n",
    "        service_provider_id = service_provider.metadata.id\n",
    "        wos_client.service_providers.delete(service_provider_id)\n",
    "        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_type = \"azure_machine_learning_service\"\n",
    "added_service_provider_result = wos_client.service_providers.add(\n",
    "        name=SERVICE_PROVIDER_NAME,\n",
    "        description=SERVICE_PROVIDER_DESCRIPTION,\n",
    "        service_type = service_type,\n",
    "        operational_space_id = \"production\",\n",
    "        credentials=AzureCredentials(\n",
    "            subscription_id= AZURE_ENGINE_CREDENTIALS['subscription_id'], \n",
    "            client_id = AZURE_ENGINE_CREDENTIALS['client_id'], \n",
    "            client_secret= AZURE_ENGINE_CREDENTIALS['client_secret'],\n",
    "            tenant = AZURE_ENGINE_CREDENTIALS['tenant']\n",
    "        ),\n",
    "        background_mode=False\n",
    "    ).result\n",
    "service_provider_id = added_service_provider_result.metadata.id\n",
    "service_provider_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id).result\n",
    "asset_deployment_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_id='***' # get the GUID for related deployment\n",
    "for model_asset_details in asset_deployment_details['resources']:\n",
    "    if model_asset_details['metadata']['guid']==deployment_id:\n",
    "        break\n",
    "model_asset_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wos_client.subscriptions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_columns = ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker']\n",
    "categorical_columns = ['CheckingStatus', 'CreditHistory', 'LoanPurpose', 'ExistingSavings', 'EmploymentDuration', 'Sex', 'OthersOnLoan', 'OwnsProperty', 'InstallmentPlans', 'Housing', 'Job', 'Telephone', 'ForeignWorker']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_asset = Asset(\n",
    "            asset_id=model_asset_details[\"entity\"][\"asset\"][\"asset_id\"],\n",
    "            name=model_asset_details[\"entity\"][\"asset\"][\"name\"],\n",
    "            url=model_asset_details[\"entity\"][\"asset\"][\"url\"],\n",
    "            asset_type=model_asset_details['entity']['asset']['asset_type'] if 'asset_type' in model_asset_details['entity']['asset'] else 'model',\n",
    "            input_data_type=InputDataType.STRUCTURED,\n",
    "            problem_type=ProblemType.BINARY_CLASSIFICATION\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointRequest\n",
    "deployment_scoring_endpoint = model_asset_details['entity']['scoring_endpoint']\n",
    "scoring_endpoint = ScoringEndpointRequest(url = model_asset_details['entity']['scoring_endpoint']['url'],request_headers = model_asset_details['entity']['scoring_endpoint']['request_headers'],\n",
    "                                                 credentials = None)  \n",
    "\n",
    "deployment = AssetDeploymentRequest(\n",
    "    deployment_id=model_asset_details['metadata']['guid'],\n",
    "    url=model_asset_details['metadata']['url'],\n",
    "    name=model_asset_details['entity']['name'],\n",
    "    description=model_asset_details['entity']['description'],\n",
    "    deployment_type=model_asset_details['entity']['type'],\n",
    "    scoring_endpoint = scoring_endpoint\n",
    ") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data_reference = TrainingDataReference(type='cos',\n",
    "                                              location=COSTrainingDataReferenceLocation(bucket = BUCKET_NAME,\n",
    "                                                                                        file_name = training_data_file_name),\n",
    "                                              connection=COSTrainingDataReferenceConnection(\n",
    "                                                                        resource_instance_id= COS_RESOURCE_CRN,\n",
    "                                                                        url= COS_ENDPOINT,\n",
    "                                                                        api_key= COS_API_KEY_ID,\n",
    "                                                                        iam_url=IAM_URL)\n",
    "                                               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asset_properties = AssetPropertiesRequest(\n",
    "        label_column=\"Risk\",\n",
    "        prediction_field='Scored Labels',\n",
    "        probability_fields=['Scored Probabilities'],\n",
    "        training_data_reference=training_data_reference,\n",
    "        training_data_schema=None,\n",
    "        input_data_schema=None,\n",
    "        output_data_schema=None,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subscription_details = wos_client.subscriptions.add(\n",
    "        data_mart_id=data_mart_id,\n",
    "        service_provider_id=service_provider_id,\n",
    "        asset=azure_asset,\n",
    "        deployment=deployment,\n",
    "        asset_properties=asset_properties,\n",
    "        background_mode=False\n",
    ").result\n",
    "subscription_id = subscription_details.metadata.id\n",
    "subscription_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(5)\n",
    "payload_data_set_id = None\n",
    "payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if payload_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", payload_data_set_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement the wrapper\n",
    "\n",
    "Once you've tested the deployment, let's implement the wrapper that will:\n",
    "* call the original scoring endpoint (credit risk)\n",
    "* conver the scoring request and response to the OpenScale format\n",
    "* store converted request and response as payload records in OpenScale data mart\n",
    "\n",
    "\n",
    "To build the correct environment for ACI, provide the following:\n",
    "* A scoring script to show how to use the scoring wrapper\n",
    "* An environment file to show what packages need to be installed\n",
    "* A configuration file to build the ACI\n",
    "\n",
    "\n",
    "### Create scoring script\n",
    "\n",
    "Create the scoring script, called score.py, used by the web service call to show how to use the deployment.\n",
    "\n",
    "You must include two required functions into the scoring script:\n",
    "* The `init()` function, which typically loads the required credentials into a global object. This function is run only once when the Docker container is started. \n",
    "\n",
    "* The `run(input_data)` function uses the original scoring endpoint to predict a value based on the input data. Next it makes format conversions and finally stored payload records in OpenScale data mart.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ACTION: You need to update below score.py content by changing the following lines:\n",
    "\n",
    "- `scoring_endpoint` - PUT your scoring endpoint there\n",
    "- `scoring_headers` - PUT your scoring header there\n",
    "- `openscale_credentials` - PUT your OpenScale credentials there\n",
    "- `openscale_subscription_uid` - PUT uid of created subscription there\n",
    "- you may also need to modify conversion methods to fit your custom format of payloads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring endpoint wrapper code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile score.py\n",
    "import json\n",
    "import numpy as np\n",
    "import os\n",
    "import time\n",
    "import requests\n",
    "import uuid\n",
    "\n",
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n",
    "from ibm_watson_openscale import APIClient\n",
    "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
    "from ibm_watson_openscale.supporting_classes.enums import *\n",
    "\n",
    "\n",
    "def convert_user_input_2_openscale(input_data):\n",
    "    users_records = input_data['input']\n",
    "    openscale_fields = list(users_records[0])\n",
    "    openscale_values = [[rec[k] for k in openscale_fields] for rec in users_records] \n",
    "\n",
    "    return {'fields':openscale_fields, 'values':openscale_values}\n",
    "\n",
    "\n",
    "def convert_user_output_2_openscale(output_data):\n",
    "    output_data = json.loads(output_data)\n",
    "    users_records = output_data['output']\n",
    "    openscale_fields = list(users_records[0])\n",
    "    openscale_values = [[rec[k] for k in openscale_fields] for rec in users_records] \n",
    "\n",
    "    return {'fields':openscale_fields, 'values':openscale_values}\n",
    "\n",
    "\n",
    "def init():\n",
    "    global openscale_client\n",
    "    global cloud_api_key\n",
    "    global openscale_subscription_uid\n",
    "    global openscale_subscription\n",
    "    global scoring_endpoint\n",
    "    global scoring_headers\n",
    "    global payload_data_set_id\n",
    "    \n",
    "    \n",
    "    scoring_endpoint = '***' #deployment scoring endpoint\n",
    "    scoring_headers = {'Content-Type': 'application/json'}\n",
    "    cloud_api_key = '***' # Cloud API key\n",
    "    openscale_subscription_uid = \"***\" # subscription id\n",
    "    authenticator = IAMAuthenticator(apikey=cloud_api_key)\n",
    "    openscale_client = APIClient(authenticator=authenticator)\n",
    "    payload_data_set_id = openscale_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=openscale_subscription_uid, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "    \n",
    "    \n",
    "def run(input_data):\n",
    "   \n",
    "    # ------ CALL SCORING ENDPOINT --------------\n",
    "    if type(input_data) is str:\n",
    "        input_data = json.loads(input_data)\n",
    "\n",
    "    start_time = time.time()        \n",
    "    response = requests.post(scoring_endpoint, json=input_data, headers=scoring_headers)\n",
    "    response_time = int((time.time() - start_time)*1000)\n",
    "    output_data = response.json()\n",
    "\n",
    "    # ------ PAYLOAD COVERSION TO OPENSCALE FORMAT and LOGGING --------------\n",
    "    try:\n",
    "        \n",
    "        openscale_input = convert_user_input_2_openscale(input_data)\n",
    "        openscale_output = convert_user_output_2_openscale(output_data)  \n",
    "        \n",
    "        records_list=[PayloadRecord(\n",
    "               request=openscale_input,\n",
    "               response=openscale_output,\n",
    "               response_time=response_time\n",
    "           )]\n",
    "        openscale_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=records_list)\n",
    "        \n",
    "    \n",
    "    except Exception as e:\n",
    "        error = str(e)\n",
    "        print(error)\n",
    "    finally:\n",
    "        return output_data\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test wrapper locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import score\n",
    "score.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = score.run(scoring_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json.loads(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wos_client.data_sets.get_records_count(data_set_id=payload_data_set_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the wrapper as web service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create environment file\n",
    "\n",
    "Next, create an environment file, called myenv.yml, that specifies all of the script's package dependencies. This file is used to ensure that all of those dependencies are installed in the Docker image. This model needs `scikit-learn` and `azureml-sdk`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "set conda dependencies"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.core.conda_dependencies import CondaDependencies \n",
    "from azureml.core import Environment\n",
    "\n",
    "environment = Environment('my-sklearn-environment')\n",
    "\n",
    "environment.python.conda_dependencies = CondaDependencies.create(pip_packages=[\n",
    "    'azureml-defaults',\n",
    "    'ibm-watson-openscale'\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create configuration file\n",
    "\n",
    "Create a deployment configuration file and specify the number of CPUs and gigabyte of RAM needed for your ACI container. While it depends on your model, the default of 1 core and 1 gigabyte of RAM is usually sufficient for many models. If you feel you need more later, you would have to recreate the image and redeploy the service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "configure web service",
     "aci"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice\n",
    "\n",
    "aciconfig = AciWebservice.deploy_configuration(cpu_cores=0.5, \n",
    "                                               memory_gb=0.5, \n",
    "                                               tags={\"data\": \"german credit risk\",  \"method\" : \"scoring-endpoint-wrapper\"}, \n",
    "                                               description='Credit risk scoring endpoint wrapper with payload logging')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy in ACI\n",
    "Estimated time to complete: **about 7-8 minutes**\n",
    "\n",
    "Configure the image and deploy. The following code goes through these steps:\n",
    "\n",
    "1. Build an image using:\n",
    "   * The scoring file (`score.py`)\n",
    "   * The environment definition\n",
    "   * The model file\n",
    "1. Register that image under the workspace. \n",
    "1. Send the image to the ACI container.\n",
    "1. Start up a container in ACI using the image.\n",
    "1. Get the web service HTTP endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring endpoint creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from azureml.core import Model\n",
    "from azureml.core.webservice import Webservice\n",
    "from azureml.exceptions import WebserviceException\n",
    "from azureml.core.model import InferenceConfig\n",
    "from azureml.core.image import ContainerImage\n",
    "\n",
    "\n",
    "service_name = '***' # wrapper deployment service name\n",
    "\n",
    "# Remove any existing service under the same name.\n",
    "try:\n",
    "    Webservice(ws, service_name).delete()\n",
    "except WebserviceException:\n",
    "    pass\n",
    "\n",
    "inference_config = InferenceConfig(entry_script='score.py', environment=environment)\n",
    "\n",
    "\n",
    "service = Model.deploy(workspace=ws,\n",
    "                       name=service_name,\n",
    "                       models=[],\n",
    "                       inference_config=inference_config,\n",
    "                       deployment_config=aciconfig,\n",
    "                       overwrite=True)\n",
    "service.wait_for_deployment(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test deployed wrapper\n",
    "\n",
    "The following code goes through these steps:\n",
    "1. Send the scoring request to the web service hosted in ACI. \n",
    "2. Print the returned predictions.\n",
    "3. Check if the scoring request and response has been logged as payload records in OpenScale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_data = {\"input\":data_df.sample(50).drop(['Risk'],axis=1).to_dict('records')\n",
    "              }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice\n",
    "\n",
    "credit_risk_wrapper_deployment_name = '***' # deployed service name\n",
    "credit_risk_wrapper_endpoint = None\n",
    "\n",
    "webservices = AciWebservice.list(ws)\n",
    "for service in webservices:\n",
    "    if service.name == credit_risk_wrapper_deployment_name:\n",
    "        credit_risk_wrapper_endpoint = service.scoring_uri\n",
    "        \n",
    "print('scoring endpoint', credit_risk_wrapper_endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "headers = {'Content-Type':'application/json'}\n",
    "resp = requests.post(credit_risk_wrapper_endpoint, json=scoring_data, headers=headers)\n",
    "\n",
    "print(\"POST to url\", credit_risk_wrapper_endpoint)\n",
    "print(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check if the payload has been logged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "time.sleep(2)\n",
    "wos_client.data_sets.get_records_count(data_set_id=payload_data_set_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quality monitoring and Feedback logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(10)\n",
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"min_feedback_data_size\": 10\n",
    "}\n",
    "thresholds = [\n",
    "                {\n",
    "                    \"metric_id\": \"area_under_roc\",\n",
    "                    \"type\": \"lower_limit\",\n",
    "                    \"value\": .80\n",
    "                }\n",
    "            ]\n",
    "quality_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters,\n",
    "    thresholds=thresholds\n",
    ").result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_monitor_instance_id = quality_monitor_details.metadata.id\n",
    "quality_monitor_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm additional_feedback_data_v2.json\n",
    "!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/additional_feedback_data_v2.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feedback_dataset_id = None\n",
    "feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, \n",
    "                                                target_target_id=subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result\n",
    "#print(feedback_dataset)\n",
    "feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id\n",
    "if feedback_dataset_id is None:\n",
    "    print(\"Feedback data set not found. Please check quality monitor status.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('additional_feedback_data_v2.json') as feedback_file:\n",
    "    additional_feedback_data = json.load(feedback_file)\n",
    "\n",
    "wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# manually records scoring and store as feedback data\n",
    "\n",
    "# records = [\n",
    "#     [\"no_checking\",\"28\",\"outstanding_credit\",\"appliances\",\"5990\",\"500_to_1000\",\"greater_7\",\"5\",\"male\",\"co-applicant\",\"3\",\"car_other\",\"55\",\"none\",\"free\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"greater_200\",\"22\",\"all_credits_paid_back\",\"car_used\",\"3376\",\"less_100\",\"less_1\",\"3\",\"female\",\"none\",\"2\",\"car_other\",\"32\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "#     [\"no_checking\",\"39\",\"credits_paid_to_date\",\"vacation\",\"6434\",\"unknown\",\"greater_7\",\"5\",\"male\",\"none\",\"4\",\"car_other\",\"39\",\"none\",\"own\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"0_to_200\",\"20\",\"credits_paid_to_date\",\"furniture\",\"2442\",\"less_100\",\"unemployed\",\"3\",\"female\",\"none\",\"1\",\"real_estate\",\"42\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "#     [\"greater_200\",\"4\",\"all_credits_paid_back\",\"education\",\"4206\",\"less_100\",\"unemployed\",\"1\",\"female\",\"none\",\"3\",\"savings_insurance\",\"27\",\"none\",\"own\",\"1\",\"management_self-employed\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "#     [\"greater_200\",\"23\",\"credits_paid_to_date\",\"car_used\",\"2963\",\"greater_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"46\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"none\",\"yes\",\"Risk\"],\n",
    "#     [\"no_checking\",\"31\",\"prior_payments_delayed\",\"vacation\",\"2673\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"2\",\"real_estate\",\"35\",\"stores\",\"rent\",\"1\",\"skilled\",\"2\",\"none\",\"yes\",\"Risk\"],\n",
    "#     [\"no_checking\",\"37\",\"prior_payments_delayed\",\"other\",\"6971\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"3\",\"savings_insurance\",\"54\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"0_to_200\",\"39\",\"prior_payments_delayed\",\"appliances\",\"5685\",\"100_to_500\",\"1_to_4\",\"4\",\"female\",\"none\",\"2\",\"unknown\",\"37\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"no_checking\",\"38\",\"prior_payments_delayed\",\"appliances\",\"4990\",\"500_to_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"50\",\"bank\",\"own\",\"2\",\"unemployed\",\"2\",\"yes\",\"yes\",\"Risk\"]]\n",
    "\n",
    "# fields = feature_columns.copy()\n",
    "# fields.append('Risk')\n",
    "\n",
    "# payload_scoring =  [{\"fields\": fields, \"values\": records}]\n",
    "# payload_scoring\n",
    "\n",
    "# wos_client.data_sets.store_records(feedback_dataset_id, request_body=payload_scoring, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(5)\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fairness, Drift and Explainability configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"features\": [\n",
    "        {\"feature\": \"Sex\",\n",
    "         \"majority\": ['male'],\n",
    "         \"minority\": ['female'],\n",
    "         \"threshold\": 0.95\n",
    "         },\n",
    "        {\"feature\": \"Age\",\n",
    "         \"majority\": [[26, 75]],\n",
    "         \"minority\": [[18, 25]],\n",
    "         \"threshold\": 0.95\n",
    "         }\n",
    "    ],\n",
    "    \"favourable_class\": [\"No Risk\"],\n",
    "    \"unfavourable_class\": [\"Risk\"],\n",
    "    \"min_records\": 50\n",
    "}\n",
    "\n",
    "fairness_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
    "    target=target,\n",
    "    parameters=parameters).result\n",
    "fairness_monitor_instance_id =fairness_monitor_details.metadata.id\n",
    "fairness_monitor_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drift configuration\n",
    "\n",
    "- Drift requires a trained model to be uploaded manually for Azure. You can train, create and download a drift detection model using template given ( check for Drift detection model generation) [here](https://github.com/IBM-Watson/aios-data-distribution/blob/master/training_statistics_notebook.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf creditrisk_azure_drift_detection_model.tar.gz\n",
    "!wget -O creditrisk_azure_drift_detection_model.tar.gz https://github.com/IBM/watson-openscale-samples/blob/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz?raw=true "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wos_client.monitor_instances.upload_drift_model(\n",
    "        model_path='creditrisk_azure_drift_detection_model.tar.gz',\n",
    "        data_mart_id=data_mart_id,\n",
    "        subscription_id=subscription_id\n",
    "     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == \"drift\" and monitor_instance.entity.target.target_id == subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"min_samples\": 50,\n",
    "    \"drift_threshold\": 0.1,\n",
    "    \"train_drift_model\": False,\n",
    "    \"enable_model_drift\": True,\n",
    "    \"enable_data_drift\": True\n",
    "}\n",
    "\n",
    "drift_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "drift_monitor_instance_id = drift_monitor_details.metadata.id\n",
    "drift_monitor_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "drift_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(5)\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explainability configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"enabled\": True\n",
    "}\n",
    "explainability_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explainability_monitor_id = explainability_details.metadata.id\n",
    "explainability_monitor_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pl_records_resp = wos_client.data_sets.get_list_of_records(data_set_id=payload_data_set_id, limit=1, offset=0).result\n",
    "scoring_ids = [pl_records_resp[\"records\"][0][\"entity\"][\"values\"][\"scoring_id\"]]\n",
    "print(\"Running explanations on scoring IDs: {}\".format(scoring_ids))\n",
    "explanation_types = [\"lime\", \"contrastive\"]\n",
    "result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types, subscription_id=subscription_id).result\n",
    "print(result)\n",
    "explanation_task_id=result.to_dict()['metadata']['explanation_task_ids'][0]\n",
    "explanation_task_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wos_client.monitor_instances.get_explanation_tasks(explanation_task_id=explanation_task_id).result.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations\n",
    "\n",
    "You have finished the tutorial for IBM Watson OpenScale and Azure Machine Learning Studio. You can now view the [OpenScale Dashboard](https://aiopenscale.cloud.ibm.com/). Click on the tile for the German Credit Azure model to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window."
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "roastala"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "msauthor": "sgilley",
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}