IBM Cloud/Azure/notebooks/ML Studio/AI OpenScale and Azure ML Studio Engine.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Azure Machine Learning Studio engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows how to monitor the model deployed on Azure Machine Learning Studio using Watson OpenScale python sdk.\n",
    "\n",
    "**Note:** This notebook works correctly with kernel **`IBM Runtime 22.1 on Python 3.9 XS`** if using IBM Watson Studio or else use standard Python 3.9 runtime."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contents\n",
    "\n",
    "1. [Setup](#setup)\n",
    "2. [Binding machine learning engine](#binding)\n",
    "3. [Subscriptions](#subsciption)\n",
    "4. [Performance monitor, scoring and payload logging](#scoring)\n",
    "5. [Quality monitor and feedback logging](#feedback)\n",
    "6. [Fairness, Drift monitor and explainability](#monitors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "# 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample model creation using [Azure Machine Learning Studio](https://studio.azureml.net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Copy [credit risk experiment](https://gallery.cortanaintelligence.com/Experiment/German-credit-risk-created-on-1-9-2019) from Azure ML Studio Gallery \n",
    "- Run experiment to train a model\n",
    "- Create (deploy) web service (new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** Classic web services are not supported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation and authentication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --upgrade --upgrade \"ibm-watson-openscale~=3.0.34\" --no-cache | tail -n 1\n",
    "!pip install --upgrade lime --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Action:** Restart the kernel (Kernel->Restart)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ACTION: Get Watson OpenScale `apikey`\n",
    "\n",
    "How to install IBM Cloud (bluemix) console: [instruction](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
    "\n",
    "How to get api key using bluemix console:\n",
    "```\n",
    "bx login --sso\n",
    "bx iam api-key-create 'my_key'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "CLOUD_API_KEY = '***'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "DB_CREDENTIALS=None\n",
    "#DB_CREDENTIALS= {\"hostname\":\"\",\"username\":\"\",\"password\":\"\",\"database\":\"\",\"port\":\"\",\"ssl\":True,\"sslmode\":\"\",\"certificate_base64\":\"\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "SCHEMA_NAME = 'azure_dm'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.0.39'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n",
    "\n",
    "from ibm_watson_openscale import *\n",
    "from ibm_watson_openscale.supporting_classes.enums import *\n",
    "from ibm_watson_openscale.supporting_classes import *\n",
    "\n",
    "\n",
    "authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n",
    "#authenticator = BearerTokenAuthenticator(bearer_token=IAM_TOKEN) ## uncomment if using IAM token\n",
    "wos_client = APIClient(authenticator=authenticator)\n",
    "wos_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download and preview training data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-07-15 16:46:55--  https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/credit_risk_training.csv\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8002::154, 2606:50c0:8003::154, 2606:50c0:8000::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8002::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 694222 (678K) [text/plain]\n",
      "Saving to: ‘credit_risk_training.csv’\n",
      "\n",
      "credit_risk_trainin 100%[===================>] 677.95K   562KB/s    in 1.2s    \n",
      "\n",
      "2024-07-15 16:46:57 (562 KB/s) - ‘credit_risk_training.csv’ saved [694222/694222]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -rf credit_risk_training.csv\n",
    "!wget \"https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/credit_risk_training.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "training_data_pd = pd.read_csv(\n",
    "        \"credit_risk_training.csv\",\n",
    "        dtype={'LoanDuration': int, 'LoanAmount': int, 'InstallmentPercent': int, 'CurrentResidenceDuration': int, 'Age': int, 'ExistingCreditsCount': int, 'Dependents': int})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CheckingStatus</th>\n",
       "      <th>LoanDuration</th>\n",
       "      <th>CreditHistory</th>\n",
       "      <th>LoanPurpose</th>\n",
       "      <th>LoanAmount</th>\n",
       "      <th>ExistingSavings</th>\n",
       "      <th>EmploymentDuration</th>\n",
       "      <th>InstallmentPercent</th>\n",
       "      <th>Sex</th>\n",
       "      <th>OthersOnLoan</th>\n",
       "      <th>...</th>\n",
       "      <th>OwnsProperty</th>\n",
       "      <th>Age</th>\n",
       "      <th>InstallmentPlans</th>\n",
       "      <th>Housing</th>\n",
       "      <th>ExistingCreditsCount</th>\n",
       "      <th>Job</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>Telephone</th>\n",
       "      <th>ForeignWorker</th>\n",
       "      <th>Risk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0_to_200</td>\n",
       "      <td>31</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>other</td>\n",
       "      <td>1889</td>\n",
       "      <td>100_to_500</td>\n",
       "      <td>less_1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>32</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>less_0</td>\n",
       "      <td>18</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>car_new</td>\n",
       "      <td>462</td>\n",
       "      <td>less_100</td>\n",
       "      <td>1_to_4</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>37</td>\n",
       "      <td>stores</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>less_0</td>\n",
       "      <td>15</td>\n",
       "      <td>prior_payments_delayed</td>\n",
       "      <td>furniture</td>\n",
       "      <td>250</td>\n",
       "      <td>less_100</td>\n",
       "      <td>1_to_4</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>real_estate</td>\n",
       "      <td>28</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0_to_200</td>\n",
       "      <td>28</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>retraining</td>\n",
       "      <td>3693</td>\n",
       "      <td>less_100</td>\n",
       "      <td>greater_7</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>32</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>No Risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>no_checking</td>\n",
       "      <td>28</td>\n",
       "      <td>prior_payments_delayed</td>\n",
       "      <td>education</td>\n",
       "      <td>6235</td>\n",
       "      <td>500_to_1000</td>\n",
       "      <td>greater_7</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>...</td>\n",
       "      <td>unknown</td>\n",
       "      <td>57</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "      <td>Risk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  CheckingStatus  LoanDuration           CreditHistory LoanPurpose  \\\n",
       "0       0_to_200            31    credits_paid_to_date       other   \n",
       "1         less_0            18    credits_paid_to_date     car_new   \n",
       "2         less_0            15  prior_payments_delayed   furniture   \n",
       "3       0_to_200            28    credits_paid_to_date  retraining   \n",
       "4    no_checking            28  prior_payments_delayed   education   \n",
       "\n",
       "   LoanAmount ExistingSavings EmploymentDuration  InstallmentPercent     Sex  \\\n",
       "0        1889      100_to_500             less_1                   3  female   \n",
       "1         462        less_100             1_to_4                   2  female   \n",
       "2         250        less_100             1_to_4                   2    male   \n",
       "3        3693        less_100          greater_7                   3    male   \n",
       "4        6235     500_to_1000          greater_7                   3    male   \n",
       "\n",
       "  OthersOnLoan  ...       OwnsProperty Age  InstallmentPlans Housing  \\\n",
       "0         none  ...  savings_insurance  32              none     own   \n",
       "1         none  ...  savings_insurance  37            stores     own   \n",
       "2         none  ...        real_estate  28              none     own   \n",
       "3         none  ...  savings_insurance  32              none     own   \n",
       "4         none  ...            unknown  57              none     own   \n",
       "\n",
       "  ExistingCreditsCount      Job Dependents  Telephone ForeignWorker     Risk  \n",
       "0                    1  skilled          1       none           yes  No Risk  \n",
       "1                    2  skilled          1       none           yes  No Risk  \n",
       "2                    2  skilled          1        yes            no  No Risk  \n",
       "3                    1  skilled          1       none           yes  No Risk  \n",
       "4                    2  skilled          1       none           yes     Risk  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data_pd.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save training data to IBM Cloud Object Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In next cells, you will need to paste some credentials to Cloud Object Storage. If you haven't worked with COS yet please visit getting started with COS tutorial. You can find COS_API_KEY_ID and COS_RESOURCE_CRN variables in Service Credentials in menu of your COS instance. Used COS Service Credentials must be created with Role parameter set as Writer. Later training data file will be loaded to the bucket of your instance and used as training refecence in subsription. COS_ENDPOINT variable can be found in Endpoint field of the menu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "COS_API_KEY_ID = \"***\"\n",
    "COS_RESOURCE_CRN = \"***\" # eg \"crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003abfb5d29761c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::\"\n",
    "COS_ENDPOINT = \"***\" # Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"***\" #example: \"credit-risk-training-data\"\n",
    "training_data_file_name=\"credit_risk_training.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ibm_boto3\n",
    "from ibm_botocore.client import Config, ClientError\n",
    "\n",
    "cos_client = ibm_boto3.resource(\"s3\",\n",
    "    ibm_api_key_id=COS_API_KEY_ID,\n",
    "    ibm_service_instance_id=COS_RESOURCE_CRN,\n",
    "    ibm_auth_endpoint=\"https://iam.bluemix.net/oidc/token\",\n",
    "    config=Config(signature_version=\"oauth\"),\n",
    "    endpoint_url=COS_ENDPOINT\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(training_data_file_name, \"rb\") as file_data:\n",
    "    cos_client.Object(BUCKET_NAME, training_data_file_name).upload_fileobj(\n",
    "        Fileobj=file_data\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataMart setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data Marts</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>description</th><th style='border: 1px solid #dddddd'>internal_database</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>AIOSFASTPATH-80E6093F-5ACF-4EB7-9DA6-7BA9BF56A929</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>True</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:07.089000+00:00</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.data_marts.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using existing datamart 80e6093f-5acf-4eb7-9da6-7ba9bf56a929\n"
     ]
    }
   ],
   "source": [
    "data_marts = wos_client.data_marts.list().result.data_marts\n",
    "if len(data_marts) == 0:\n",
    "    if DB_CREDENTIALS is not None:\n",
    "        if SCHEMA_NAME is None: \n",
    "            print(\"Please specify the SCHEMA_NAME and rerun the cell\")\n",
    "\n",
    "        print('Setting up external datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\",\n",
    "                database_configuration=DatabaseConfigurationRequest(\n",
    "                  database_type=DatabaseType.POSTGRESQL,\n",
    "                    credentials=PrimaryStorageCredentialsLong(\n",
    "                        hostname=DB_CREDENTIALS['hostname'],\n",
    "                        username=DB_CREDENTIALS['username'],\n",
    "                        password=DB_CREDENTIALS['password'],\n",
    "                        db=DB_CREDENTIALS['database'],\n",
    "                        port=DB_CREDENTIALS['port'],\n",
    "                        ssl=True,\n",
    "                        sslmode=DB_CREDENTIALS['sslmode'],\n",
    "                        certificate_base64=DB_CREDENTIALS['certificate_base64']\n",
    "                    ),\n",
    "                    location=LocationSchemaName(\n",
    "                        schema_name= SCHEMA_NAME\n",
    "                    )\n",
    "                )\n",
    "             ).result\n",
    "    else:\n",
    "        print('Setting up internal datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\", \n",
    "                internal_database = True).result\n",
    "        \n",
    "    data_mart_id = added_data_mart_result.metadata.id\n",
    "    \n",
    "else:\n",
    "    data_mart_id=data_marts[0].metadata.id\n",
    "    print('Using existing datamart {}'.format(data_mart_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"binding\"></a>\n",
    "# 2. Bind machine learning engines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bind  `Azure` machine learning studio\n",
    "\n",
    "Provide credentials using following fields:\n",
    "- `client_id`\n",
    "- `client_secret`\n",
    "- `subscription_id`\n",
    "- `tenant`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "AZURE_ENGINE_CREDENTIALS = {\n",
    "    \"client_id\": \"***\",\n",
    "    \"client_secret\": \"***\",\n",
    "    \"subscription_id\": \"***\",\n",
    "    \"tenant\": \"***\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "SERVICE_PROVIDER_NAME = \"Azure Machine Learning\"\n",
    "SERVICE_PROVIDER_DESCRIPTION = \"Added by Azure tutorial WOS notebook.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "service_providers = wos_client.service_providers.list().result.service_providers\n",
    "for service_provider in service_providers:\n",
    "    service_instance_name = service_provider.entity.name\n",
    "    if service_instance_name == SERVICE_PROVIDER_NAME:\n",
    "        service_provider_id = service_provider.metadata.id\n",
    "        wos_client.service_providers.delete(service_provider_id)\n",
    "        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=================================================================================\n",
      "\n",
      " Waiting for end of adding service provider db2618f1-ef2b-4251-a22c-bd347702672d \n",
      "\n",
      "=================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "-----------------------------------------------\n",
      " Successfully finished adding service provider \n",
      "-----------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "added_service_provider_result = wos_client.service_providers.add(\n",
    "        name=SERVICE_PROVIDER_NAME,\n",
    "        description=SERVICE_PROVIDER_DESCRIPTION,\n",
    "        service_type=ServiceTypes.AZURE_MACHINE_LEARNING,\n",
    "        operational_space_id = \"production\",\n",
    "        credentials=AzureCredentials(\n",
    "            subscription_id= AZURE_ENGINE_CREDENTIALS['subscription_id'], \n",
    "            client_id = AZURE_ENGINE_CREDENTIALS['client_id'], \n",
    "            client_secret= AZURE_ENGINE_CREDENTIALS['client_secret'],\n",
    "            tenant = AZURE_ENGINE_CREDENTIALS['tenant']\n",
    "        ),\n",
    "        background_mode=False\n",
    "    ).result\n",
    "service_provider_id = added_service_provider_result.metadata.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Service Providers</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>instance_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>name</th><th style='border: 1px solid #dddddd'>service_type</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>Azure Machine Learning</td><td style='border: 1px solid #dddddd'>azure_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-15 11:18:22.795000+00:00</td><td style='border: 1px solid #dddddd'>db2618f1-ef2b-4251-a22c-bd347702672d</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WML - Training Notebooks</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-15 10:05:51.934000+00:00</td><td style='border: 1px solid #dddddd'>06a2bb81-8027-4b41-bc7a-5a24f323fbef</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>AWS Machine Learning</td><td style='border: 1px solid #dddddd'>amazon_sagemaker</td><td style='border: 1px solid #dddddd'>2024-07-11 09:29:39.272000+00:00</td><td style='border: 1px solid #dddddd'>4fa5625b-5401-4a31-a423-d480ea184cda</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>OpenScale Headless Service Provider</td><td style='border: 1px solid #dddddd'>custom_machine_learning</td><td style='border: 1px solid #dddddd'>2024-07-05 13:55:20.532000+00:00</td><td style='border: 1px solid #dddddd'>0bf0f943-24cf-487e-b971-4a6f586effc0</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>aWatson Machine Learning V2_amzon</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-06-06 06:03:03.459000+00:00</td><td style='border: 1px solid #dddddd'>84265de0-8f4b-4257-8743-7c2525e86b00</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>Watson Machine Learning V2</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-06-02 08:24:57.362000+00:00</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WOS ExpressPath WML pre_production binding</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:28.464000+00:00</td><td style='border: 1px solid #dddddd'>19fc94cc-9ed2-4ac6-a98e-1acf1a9e729e</td></tr><tr><td style='border: 1px solid #dddddd'>89cbf463-87d9-4032-9c0c-e371dd68d156</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>WOS ExpressPath WML production binding</td><td style='border: 1px solid #dddddd'>watson_machine_learning</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:12.960000+00:00</td><td style='border: 1px solid #dddddd'>7ef8cc5f-65b1-4f83-9d1b-d2d508420958</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.service_providers.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id).result\n",
    "asset_deployment_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "## select your model deployment GUID and get model asset details\n",
    "\n",
    "deployment_id=''\n",
    "for model_asset_details in asset_deployment_details['resources']:\n",
    "    if model_asset_details['metadata']['guid']==deployment_id:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sample scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Results\": {\n",
      "    \"output1\": [\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"19\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"business\",\n",
      "        \"LoanAmount\": \"1867\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"42\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"unskilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.000162622847710736\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import time\n",
    "import json\n",
    "\n",
    "scoring_url = model_asset_details['entity']['scoring_endpoint']['url']\n",
    "\n",
    "data = {\n",
    "            \"Inputs\": {\n",
    "                \"input1\":\n",
    "                    [{'CheckingStatus': 'no_checking',\n",
    "                      'LoanDuration': 19,\n",
    "                      'CreditHistory': 'credits_paid_to_date',\n",
    "                      'LoanPurpose': 'business',\n",
    "                      'LoanAmount': 1867,\n",
    "                      'ExistingSavings': '100_to_500',\n",
    "                      'EmploymentDuration': '1_to_4',\n",
    "                      'InstallmentPercent': 4,\n",
    "                      'Sex': 'male',\n",
    "                      'OthersOnLoan': 'none',\n",
    "                      'CurrentResidenceDuration': 3,\n",
    "                      'OwnsProperty': 'car_other',\n",
    "                      'Age': 42,\n",
    "                      'InstallmentPlans': 'stores',\n",
    "                      'Housing': 'own',\n",
    "                      'ExistingCreditsCount': 2,\n",
    "                      'Job': 'unskilled',\n",
    "                      'Dependents': 1,\n",
    "                      'Telephone': 'none',\n",
    "                      'ForeignWorker': 'yes'}]\n",
    "            },\n",
    "            \"GlobalParameters\": {\n",
    "            }\n",
    "        }\n",
    "\n",
    "body = str.encode(json.dumps(data))\n",
    "\n",
    "\n",
    "token = model_asset_details['entity']['scoring_endpoint']['credentials']['token']\n",
    "headers = model_asset_details['entity']['scoring_endpoint']['request_headers']\n",
    "headers['Authorization'] = ('Bearer ' + token)\n",
    "\n",
    "start_time = time.time()\n",
    "response = requests.post(url=scoring_url, data=body, headers=headers)\n",
    "response_time = int(time.time() - start_time)*1000\n",
    "result = response.json()\n",
    "\n",
    "print(json.dumps(result, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"subsciption\"></a>\n",
    "# 3. Subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add subscriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List available deployments.\n",
    "\n",
    "**Note:** Depending on number of assets it may take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Subscriptions</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_id</th><th style='border: 1px solid #dddddd'>asset_type</th><th style='border: 1px solid #dddddd'>asset_name</th><th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>deployment_name</th><th style='border: 1px solid #dddddd'>service_provider_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>3c89d675-9c16-4f02-8eb1-a7744b6b1e4b</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Training stats</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>Spark German Risk Deployment - Training stats</td><td style='border: 1px solid #dddddd'>06a2bb81-8027-4b41-bc7a-5a24f323fbef</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-15 10:23:38.109000+00:00</td><td style='border: 1px solid #dddddd'>d43392d4-763e-4e90-b8d2-c20c29d3939e</td></tr><tr><td style='border: 1px solid #dddddd'>3c89d675-9c16-4f02-8eb1-a7744b6b1e4b</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Training stats</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>Spark German Risk Deployment - Training stats</td><td style='border: 1px solid #dddddd'>06a2bb81-8027-4b41-bc7a-5a24f323fbef</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-15 10:16:10.364000+00:00</td><td style='border: 1px solid #dddddd'>81d82d93-0025-44b8-af81-b0eeaed2e784</td></tr><tr><td style='border: 1px solid #dddddd'>3c89d675-9c16-4f02-8eb1-a7744b6b1e4b</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Training stats</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>Spark German Risk Deployment - Training stats</td><td style='border: 1px solid #dddddd'>06a2bb81-8027-4b41-bc7a-5a24f323fbef</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-15 10:12:44.840000+00:00</td><td style='border: 1px solid #dddddd'>6e0eade8-3741-4985-82c6-ee7ccc209646</td></tr><tr><td style='border: 1px solid #dddddd'>3c89d675-9c16-4f02-8eb1-a7744b6b1e4b</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Training stats</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>5b644de6-5b83-4930-ac85-c600b2c0ac4f</td><td style='border: 1px solid #dddddd'>Spark German Risk Deployment - Training stats</td><td style='border: 1px solid #dddddd'>06a2bb81-8027-4b41-bc7a-5a24f323fbef</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-15 10:06:12.336000+00:00</td><td style='border: 1px solid #dddddd'>c015cc58-7d78-4311-8f94-0bd46cc302f8</td></tr><tr><td style='border: 1px solid #dddddd'>98783e76b0aedefc06c70bd74fa2356e</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Credit-risk-linear-learner-2021-06-24-05-32</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>cc7acf711b19c689d67e7b2a01b88ca2</td><td style='border: 1px solid #dddddd'>Credit-risk-endpoint-scoring-2021-06-24-05-32</td><td style='border: 1px solid #dddddd'>4fa5625b-5401-4a31-a423-d480ea184cda</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-11 09:30:10.069000+00:00</td><td style='border: 1px solid #dddddd'>95e480af-2fec-43ef-b34e-b3a24572b35d</td></tr><tr><td style='border: 1px solid #dddddd'>d4259f6b-e5a1-4a33-a862-d2788e5289ef</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>[asset] GCR Headless Subscription</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>aa8bd973-d680-496b-923d-f9bcbcc0bc7a</td><td style='border: 1px solid #dddddd'>[asset] GCR Headless Subscription</td><td style='border: 1px solid #dddddd'>0bf0f943-24cf-487e-b971-4a6f586effc0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-05 13:57:01.523000+00:00</td><td style='border: 1px solid #dddddd'>345f4646-871a-4e0c-b30f-abcf9a83ea71</td></tr><tr><td style='border: 1px solid #dddddd'>ad45394c-0cfb-403c-81f6-a20f44cded7a</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Spark German Risk Model - Final</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>8db12c5c-94cf-486e-937a-6f17dadf8a56</td><td style='border: 1px solid #dddddd'>New - Spark German Risk Deployment - Final</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-07-04 09:15:28.269000+00:00</td><td style='border: 1px solid #dddddd'>59622413-a37f-4e76-a4d5-46967f3cf12b</td></tr><tr><td style='border: 1px solid #dddddd'>43870a88-5fb5-425a-9099-078b0469dd2a</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Amazon Mobile Review Classifier</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>8edfc920-a4bc-47cd-83fd-9e9674476dec</td><td style='border: 1px solid #dddddd'>Amazon Mobile Review Classifier deployment2</td><td style='border: 1px solid #dddddd'>84265de0-8f4b-4257-8743-7c2525e86b00</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-06-06 06:04:06.732000+00:00</td><td style='border: 1px solid #dddddd'>7828e5f6-9cc7-4cc6-b20b-7bf844c38ccc</td></tr><tr><td style='border: 1px solid #dddddd'>699a18fc-1d52-4a1b-9a25-87d9362cd3dc</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>Scikit German Risk Model WML V4</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>2de6645d-c6a9-42ae-a884-4e2bb646ad33</td><td style='border: 1px solid #dddddd'>Scikit German Risk Deployment WML V4-1</td><td style='border: 1px solid #dddddd'>e13bfbbb-e639-4875-bbd5-6d2de8e834d0</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-06-02 08:26:51.212000+00:00</td><td style='border: 1px solid #dddddd'>91a07f4c-50b6-426d-93da-94438a7edc9a</td></tr><tr><td style='border: 1px solid #dddddd'>a7fc7ed9-f000-4f27-988a-54fbc8e397e6</td><td style='border: 1px solid #dddddd'>model</td><td style='border: 1px solid #dddddd'>GermanCreditRiskModelChallenger</td><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>38059feb-e4c5-4037-90ea-a18f77a8ec5b</td><td style='border: 1px solid #dddddd'>GermanCreditRiskModelChallenger</td><td style='border: 1px solid #dddddd'>19fc94cc-9ed2-4ac6-a98e-1acf1a9e729e</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>2024-05-16 06:09:55.832000+00:00</td><td style='border: 1px solid #dddddd'>81044f9c-27a8-42f2-90be-9cef8b33fe39</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.subscriptions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "feature_columns = ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker']\n",
    "categorical_columns = ['CheckingStatus', 'CreditHistory', 'LoanPurpose', 'ExistingSavings', 'EmploymentDuration', 'Sex', 'OthersOnLoan', 'OwnsProperty', 'InstallmentPlans', 'Housing', 'Job', 'Telephone', 'ForeignWorker']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointCredentialsAzureScoringEndpointCredentials,ScoringEndpointRequest\n",
    "def get_output_schema_fields_for_azure(azure_ouput_schema, target_field, target_role = \"decoded-target\", field_type = str, is_nullable = True):\n",
    "        fields = []\n",
    "        found = False\n",
    "        for field in azure_ouput_schema['fields']:\n",
    "            if field['name']==target_field:\n",
    "                metadata = {}\n",
    "                if \"metadata\" in field:\n",
    "                    metadata = field[\"metadata\"]\n",
    "                metadata[\"modeling_role\"]=target_role    \n",
    "                field[\"metadata\"] = metadata\n",
    "                found = True\n",
    "    \n",
    "            spark_field = SparkStructFieldObject(\n",
    "                name=field['name'],\n",
    "                type=field['type'],\n",
    "                nullable=field['nullable'],\n",
    "            )\n",
    "            if \"metadata\" in field:\n",
    "                spark_field.metadata = field[\"metadata\"]\n",
    "                \n",
    "            fields.append(spark_field)   \n",
    "        \n",
    "        if found is False:\n",
    "            metadata = {}\n",
    "            metadata[\"modeling_role\"] = target_role\n",
    "            fields.append(SparkStructFieldObject(\n",
    "                name=target_field,\n",
    "                type=field_type,\n",
    "                nullable=is_nullable,\n",
    "                metadata = metadata\n",
    "            )) \n",
    "        return fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_asset = Asset(\n",
    "            asset_id=model_asset_details[\"entity\"][\"asset\"][\"asset_id\"],\n",
    "            name=model_asset_details[\"entity\"][\"asset\"][\"name\"],\n",
    "            url=model_asset_details[\"entity\"][\"asset\"][\"url\"],\n",
    "            asset_type=AssetTypes.MODEL,\n",
    "            input_data_type=InputDataType.STRUCTURED,\n",
    "            problem_type=ProblemType.BINARY_CLASSIFICATION,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_scoring_endpoint = model_asset_details['entity']['scoring_endpoint']\n",
    "scoring_endpoint = ScoringEndpointRequest(url = model_asset_details['entity']['scoring_endpoint']['url']\n",
    "                                          ,request_headers = model_asset_details['entity']['scoring_endpoint']['request_headers'],\n",
    "                                          credentials= ScoringEndpointCredentialsAzureScoringEndpointCredentials(model_asset_details['entity']['scoring_endpoint']['credentials']['token']) )\n",
    "\n",
    "deployment = AssetDeploymentRequest(\n",
    "    deployment_id=model_asset_details['metadata']['guid'],\n",
    "    url=model_asset_details['metadata']['url'],\n",
    "    name=model_asset_details['entity']['name'],\n",
    "    description=model_asset_details['entity']['description'],\n",
    "    deployment_type=model_asset_details['entity']['type'],\n",
    "    scoring_endpoint = scoring_endpoint\n",
    ") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'type': 'struct',\n",
       " 'fields': [{'name': 'CheckingStatus', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'LoanDuration', 'type': 'integer', 'nullable': False},\n",
       "  {'name': 'CreditHistory', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'LoanPurpose', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'LoanAmount', 'type': 'integer', 'nullable': False},\n",
       "  {'name': 'ExistingSavings', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'EmploymentDuration', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'InstallmentPercent', 'type': 'integer', 'nullable': False},\n",
       "  {'name': 'Sex', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'OthersOnLoan', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'CurrentResidenceDuration', 'type': 'integer', 'nullable': False},\n",
       "  {'name': 'OwnsProperty', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'Age', 'type': 'integer', 'nullable': False},\n",
       "  {'name': 'InstallmentPlans', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'Housing', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'ExistingCreditsCount', 'type': 'integer', 'nullable': False},\n",
       "  {'name': 'Job', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'Dependents', 'type': 'integer', 'nullable': False},\n",
       "  {'name': 'Telephone', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'ForeignWorker', 'type': 'string', 'nullable': False},\n",
       "  {'name': 'Scored Labels',\n",
       "   'type': 'string',\n",
       "   'nullable': False,\n",
       "   'metadata': {'modeling_role': 'decoded-target'}},\n",
       "  {'name': 'Scored Probabilities',\n",
       "   'type': 'double',\n",
       "   'nullable': False,\n",
       "   'metadata': {'modeling_role': 'class_probability'}},\n",
       "  {'name': 'ProbabilityVector',\n",
       "   'type': {'type': 'array', 'elementType': 'double', 'containsNull': False},\n",
       "   'nullable': False,\n",
       "   'metadata': {'modeling_role': 'probability'}}]}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data_reference = TrainingDataReference(type='cos',\n",
    "                                              location=COSTrainingDataReferenceLocation(bucket = BUCKET_NAME,\n",
    "                                                                                        file_name = training_data_file_name),\n",
    "                                              connection=COSTrainingDataReferenceConnection(\n",
    "                                                                        resource_instance_id= COS_RESOURCE_CRN,\n",
    "                                                                        url= COS_ENDPOINT,\n",
    "                                                                        api_key= COS_API_KEY_ID,\n",
    "                                                                        iam_url=IAM_URL)\n",
    "                                               )\n",
    "os = model_asset_details['entity']['asset_properties']['output_data_schema']\n",
    "input_s= model_asset_details['entity']['asset_properties']['input_data_schema']\n",
    "input_schema = SparkStruct(\n",
    "        type=model_asset_details['entity']['asset_properties']['input_data_schema']['type'],\n",
    "        fields=model_asset_details['entity']['asset_properties']['input_data_schema']['fields']\n",
    "    )\n",
    "output_schema = SparkStruct(\n",
    "        type=model_asset_details['entity']['asset_properties']['output_data_schema']['type'],\n",
    "        fields=get_output_schema_fields_for_azure(model_asset_details['entity']['asset_properties']['output_data_schema'], target_field = \"Scored Labels\", target_role = \"decoded-target\")\n",
    "    )\n",
    "output_schema.to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "asset_properties = AssetPropertiesRequest(\n",
    "        label_column=\"Risk\",\n",
    "        prediction_field='Scored Labels',\n",
    "        probability_fields=['Scored Probabilities'],\n",
    "        training_data_reference=training_data_reference,\n",
    "        training_data_schema=None,\n",
    "        input_data_schema=input_schema,\n",
    "        output_data_schema=output_schema,\n",
    "        feature_fields=feature_columns,\n",
    "        categorical_fields=categorical_columns\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=============================================================================\n",
      "\n",
      " Waiting for end of adding subscription 1ae2f751-e220-48a9-b049-d2404311f314 \n",
      "\n",
      "=============================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "-------------------------------------------\n",
      " Successfully finished adding subscription \n",
      "-------------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'1ae2f751-e220-48a9-b049-d2404311f314'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subscription_details = wos_client.subscriptions.add(\n",
    "        data_mart_id=data_mart_id,\n",
    "        service_provider_id=service_provider_id,\n",
    "        asset=azure_asset,\n",
    "        deployment=deployment,\n",
    "        asset_properties=asset_properties,\n",
    "        background_mode=False\n",
    ").result\n",
    "subscription_id = subscription_details.metadata.id\n",
    "subscription_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"scoring\"></a>\n",
    "# 4. Scoring and payload logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score the credit risk model and measure response time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Payload data set id:  b5f897f0-20ec-46cb-9289-9e92c188603f\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(5)\n",
    "payload_data_set_id = None\n",
    "payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if payload_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", payload_data_set_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Results\": {\n",
      "    \"output1\": [\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"32\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"appliances\",\n",
      "        \"LoanAmount\": \"4902\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"33\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"free\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.325806558132172\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"11\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"5628\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"24\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0146799990907311\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import time\n",
    "import json\n",
    "\n",
    "subscription_details=wos_client.subscriptions.get(subscription_id).result.to_dict()\n",
    "scoring_url = subscription_details['entity']['deployment']['scoring_endpoint']['url']\n",
    "\n",
    "data = {\n",
    "            \"Inputs\": {\n",
    "                \"input1\":\n",
    "                    training_data_pd.sample(2).drop(['Risk'],axis=1).to_dict('records'),\n",
    "            },\n",
    "            \"GlobalParameters\": {\n",
    "            }\n",
    "        }\n",
    "\n",
    "body = str.encode(json.dumps(data))\n",
    "\n",
    "token = model_asset_details['entity']['scoring_endpoint']['credentials']['token']\n",
    "headers = subscription_details['entity']['deployment']['scoring_endpoint']['request_headers']\n",
    "headers['Authorization'] = ('Bearer ' + token)\n",
    "\n",
    "start_time = time.time()\n",
    "response = requests.post(url=scoring_url, data=body, headers=headers)\n",
    "response_time = int(time.time() - start_time)*1000\n",
    "result = response.json()\n",
    "\n",
    "print(json.dumps(result, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions to adjust payload data format for OpenScale "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_user_input_2_openscale(input_data):\n",
    "    users_records = input_data['Inputs']['input1']\n",
    "    openscale_fields = list(users_records[0])\n",
    "    openscale_values = [[rec[k] for k in openscale_fields] for rec in users_records] \n",
    "\n",
    "    return {'fields':openscale_fields, 'values':openscale_values}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "def convert_user_output_2_openscale(output_data):\n",
    "    users_records = output_data['Results']['output1']\n",
    "    openscale_fields = [\"Scored Labels\",\"Scored Probabilities\"]\n",
    "    openscale_values = [[np.double(rec[k]) if k=='Scored Probabilities' else rec[k] for k in openscale_fields ] for rec in users_records] \n",
    "\n",
    "    return {'fields':openscale_fields, 'values':openscale_values}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "openscale_input=convert_user_input_2_openscale(data)\n",
    "openscale_output=convert_user_output_2_openscale(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store the request and response in payload logging table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performing explicit payload logging.....\n",
      "Number of records in the payload logging table: 2\n"
     ]
    }
   ],
   "source": [
    "import uuid\n",
    "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
    "\n",
    "print(\"Performing explicit payload logging.....\")\n",
    "wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(\n",
    "           scoring_id=str(uuid.uuid4()),\n",
    "           request=openscale_input,\n",
    "           response=openscale_output,\n",
    "           response_time=460\n",
    ")])\n",
    "time.sleep(5)\n",
    "pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
    "print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data Set b5f897f0-20ec-46cb-9289-9e92c188603f Records</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_revision</th><th style='border: 1px solid #dddddd'>LoanAmount</th><th style='border: 1px solid #dddddd'>scoring_id</th><th style='border: 1px solid #dddddd'>InstallmentPercent</th><th style='border: 1px solid #dddddd'>ExistingSavings</th><th style='border: 1px solid #dddddd'>scoring_timestamp</th><th style='border: 1px solid #dddddd'>prediction_probability</th><th style='border: 1px solid #dddddd'>ProbabilityVector</th><th style='border: 1px solid #dddddd'>CheckingStatus</th><th style='border: 1px solid #dddddd'>LoanPurpose</th><th style='border: 1px solid #dddddd'>Scored Probabilities</th><th style='border: 1px solid #dddddd'>Job</th><th style='border: 1px solid #dddddd'>InstallmentPlans</th><th style='border: 1px solid #dddddd'>OthersOnLoan</th><th style='border: 1px solid #dddddd'>LoanDuration</th><th style='border: 1px solid #dddddd'>Age</th><th style='border: 1px solid #dddddd'>CreditHistory</th><th style='border: 1px solid #dddddd'>CurrentResidenceDuration</th><th style='border: 1px solid #dddddd'>ForeignWorker</th><th style='border: 1px solid #dddddd'>Housing</th><th style='border: 1px solid #dddddd'>ExistingCreditsCount</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>Telephone</th><th style='border: 1px solid #dddddd'>Scored Labels</th><th style='border: 1px solid #dddddd'>Sex</th><th style='border: 1px solid #dddddd'>Dependents</th><th style='border: 1px solid #dddddd'>OwnsProperty</th><th style='border: 1px solid #dddddd'>EmploymentDuration</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>4902</td><td style='border: 1px solid #dddddd'>a6fc01cd-3912-40b1-b999-97dbaeb3cf24-1</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:20:45.191Z</td><td style='border: 1px solid #dddddd'>0.674193441867828</td><td style='border: 1px solid #dddddd'>[0.674193441867828, 0.325806558132172]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>appliances</td><td style='border: 1px solid #dddddd'>0.325806558132172</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>32</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>5628</td><td style='border: 1px solid #dddddd'>a6fc01cd-3912-40b1-b999-97dbaeb3cf24-2</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:20:45.191Z</td><td style='border: 1px solid #dddddd'>0.9853200009092689</td><td style='border: 1px solid #dddddd'>[0.9853200009092689, 0.0146799990907311]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>0.0146799990907311</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>11</td><td style='border: 1px solid #dddddd'>24</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.data_sets.show_records(data_set_id=payload_data_set_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"feedback\"></a>\n",
    "# 5. Feedback logging & quality (accuracy) monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enable quality monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation fc317b7c-0d31-43f9-b517-a65b44047e9d \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(10)\n",
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"min_feedback_data_size\": 10\n",
    "}\n",
    "thresholds = [\n",
    "                {\n",
    "                    \"metric_id\": \"area_under_roc\",\n",
    "                    \"type\": \"lower_limit\",\n",
    "                    \"value\": .80\n",
    "                }\n",
    "            ]\n",
    "quality_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters,\n",
    "    thresholds=thresholds\n",
    ").result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fc317b7c-0d31-43f9-b517-a65b44047e9d'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quality_monitor_instance_id = quality_monitor_details.metadata.id\n",
    "quality_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feedback records logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feedback records are used to evaluate your model. The predicted values are compared to real values (feedback records)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: additional_feedback_data_v2.json: No such file or directory\n",
      "--2024-07-15 16:51:19--  https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/additional_feedback_data_v2.json\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8002::154, 2606:50c0:8003::154, 2606:50c0:8000::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8002::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 50890 (50K) [text/plain]\n",
      "Saving to: ‘additional_feedback_data_v2.json’\n",
      "\n",
      "additional_feedback 100%[===================>]  49.70K  --.-KB/s    in 0.03s   \n",
      "\n",
      "2024-07-15 16:51:19 (1.73 MB/s) - ‘additional_feedback_data_v2.json’ saved [50890/50890]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm additional_feedback_data_v2.json\n",
    "!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/additional_feedback_data_v2.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get feedback logging dataset ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "feedback_dataset_id = None\n",
    "feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, \n",
    "                                                target_target_id=subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result\n",
    "#print(feedback_dataset)\n",
    "feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id\n",
    "if feedback_dataset_id is None:\n",
    "    print(\"Feedback data set not found. Please check quality monitor status.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "==========================================================================================\n",
      "\n",
      " Waiting for end of storing records with request id: e7f1978e-b06a-4edd-891e-ce8345d7c25d \n",
      "\n",
      "==========================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Successfully finished storing records \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<ibm_cloud_sdk_core.detailed_response.DetailedResponse at 0x137e9bbe0>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('additional_feedback_data_v2.json') as feedback_file:\n",
    "    additional_feedback_data = json.load(feedback_file)\n",
    "wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# manually records scoring and store as feedback data\n",
    "\n",
    "# records = [\n",
    "#     [\"no_checking\",\"28\",\"outstanding_credit\",\"appliances\",\"5990\",\"500_to_1000\",\"greater_7\",\"5\",\"male\",\"co-applicant\",\"3\",\"car_other\",\"55\",\"none\",\"free\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"greater_200\",\"22\",\"all_credits_paid_back\",\"car_used\",\"3376\",\"less_100\",\"less_1\",\"3\",\"female\",\"none\",\"2\",\"car_other\",\"32\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "#     [\"no_checking\",\"39\",\"credits_paid_to_date\",\"vacation\",\"6434\",\"unknown\",\"greater_7\",\"5\",\"male\",\"none\",\"4\",\"car_other\",\"39\",\"none\",\"own\",\"2\",\"skilled\",\"2\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"0_to_200\",\"20\",\"credits_paid_to_date\",\"furniture\",\"2442\",\"less_100\",\"unemployed\",\"3\",\"female\",\"none\",\"1\",\"real_estate\",\"42\",\"none\",\"own\",\"1\",\"skilled\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "#     [\"greater_200\",\"4\",\"all_credits_paid_back\",\"education\",\"4206\",\"less_100\",\"unemployed\",\"1\",\"female\",\"none\",\"3\",\"savings_insurance\",\"27\",\"none\",\"own\",\"1\",\"management_self-employed\",\"1\",\"none\",\"yes\",\"No Risk\"],\n",
    "#     [\"greater_200\",\"23\",\"credits_paid_to_date\",\"car_used\",\"2963\",\"greater_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"46\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"none\",\"yes\",\"Risk\"],\n",
    "#     [\"no_checking\",\"31\",\"prior_payments_delayed\",\"vacation\",\"2673\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"2\",\"real_estate\",\"35\",\"stores\",\"rent\",\"1\",\"skilled\",\"2\",\"none\",\"yes\",\"Risk\"],\n",
    "#     [\"no_checking\",\"37\",\"prior_payments_delayed\",\"other\",\"6971\",\"500_to_1000\",\"1_to_4\",\"3\",\"male\",\"none\",\"3\",\"savings_insurance\",\"54\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"0_to_200\",\"39\",\"prior_payments_delayed\",\"appliances\",\"5685\",\"100_to_500\",\"1_to_4\",\"4\",\"female\",\"none\",\"2\",\"unknown\",\"37\",\"none\",\"own\",\"2\",\"skilled\",\"1\",\"yes\",\"yes\",\"Risk\"],\n",
    "#     [\"no_checking\",\"38\",\"prior_payments_delayed\",\"appliances\",\"4990\",\"500_to_1000\",\"greater_7\",\"4\",\"male\",\"none\",\"4\",\"car_other\",\"50\",\"bank\",\"own\",\"2\",\"unemployed\",\"2\",\"yes\",\"yes\",\"Risk\"]]\n",
    "\n",
    "# fields = feature_columns.copy()\n",
    "# fields.append('Risk')\n",
    "\n",
    "# payload_scoring =  [{\"fields\": fields, \"values\": records}]\n",
    "#wos_client.data_sets.store_records(feedback_dataset_id, request_body=payload_scoring, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "98"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run 27778256-b612-48bb-b04e-51d4d45ac3a1 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>fc317b7c-0d31-43f9-b517-a65b44047e9d Monitor Runs Metrics from: 2024-07-08 16:51:48.554318  till: 2024-07-15 16:51:48.554412</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>true_positive_rate</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.6060606060606061</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>area_under_roc</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.7414918414918414</td><td style='border: 1px solid #dddddd'>0.8</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>precision</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.7142857142857143</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>matthews_correlation_coefficient</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.5052636690792</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>f1_measure</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.6557377049180327</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>accuracy</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.7857142857142857</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>label_skew</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.6909336273400484</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>gini_coefficient</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.4829836829836829</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>log_loss</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.754676424586092</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr><tr><td style='border: 1px solid #dddddd'>2024-07-15 11:21:34.497000+00:00</td><td style='border: 1px solid #dddddd'>false_positive_rate</td><td style='border: 1px solid #dddddd'>9c8cec7c-ee52-4970-b206-172333acf9fe</td><td style='border: 1px solid #dddddd'>0.12307692307692308</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>['model_type:original']</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td><td style='border: 1px solid #dddddd'>27778256-b612-48bb-b04e-51d4d45ac3a1</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "time.sleep(5)\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Fairness, Drift monitoring and explanations <a id=\"monitors\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fairness monitor configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Monitor instances</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>data_mart_id</th><th style='border: 1px solid #dddddd'>status</th><th style='border: 1px solid #dddddd'>target_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>created_at</th><th style='border: 1px solid #dddddd'>id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>quality</td><td style='border: 1px solid #dddddd'>2024-07-15 11:21:07.975000+00:00</td><td style='border: 1px solid #dddddd'>fc317b7c-0d31-43f9-b517-a65b44047e9d</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>81d82d93-0025-44b8-af81-b0eeaed2e784</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>fairness</td><td style='border: 1px solid #dddddd'>2024-07-15 10:21:17.571000+00:00</td><td style='border: 1px solid #dddddd'>53c49c40-e8db-404f-82c0-3bb29275c7de</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:22.435000+00:00</td><td style='border: 1px solid #dddddd'>710c1e12-6188-4d5f-a8cc-9596f8e559e3</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>performance</td><td style='border: 1px solid #dddddd'>2024-07-15 11:20:24.211000+00:00</td><td style='border: 1px solid #dddddd'>de132bed-c9fe-4c4f-a767-ab1c396f4372</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>59622413-a37f-4e76-a4d5-46967f3cf12b</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>2024-07-04 09:15:33.364000+00:00</td><td style='border: 1px solid #dddddd'>3ac554fe-86b5-47c1-b0c8-0dbbc60973e5</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>59622413-a37f-4e76-a4d5-46967f3cf12b</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-04 09:15:34.074000+00:00</td><td style='border: 1px solid #dddddd'>8dc0726c-618d-49fe-9c9b-a4d6544f96f4</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>6e0eade8-3741-4985-82c6-ee7ccc209646</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-15 10:12:47.370000+00:00</td><td style='border: 1px solid #dddddd'>e4ed47fa-aa45-4ddd-9663-f6a91d598107</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>6e0eade8-3741-4985-82c6-ee7ccc209646</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>fairness</td><td style='border: 1px solid #dddddd'>2024-07-15 10:13:18.402000+00:00</td><td style='border: 1px solid #dddddd'>537eab0b-4dbb-460b-9e99-322ca89e8ec6</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>345f4646-871a-4e0c-b30f-abcf9a83ea71</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>mrm</td><td style='border: 1px solid #dddddd'>2024-07-05 14:08:34.802000+00:00</td><td style='border: 1px solid #dddddd'>2f9e942b-5170-419f-989c-bc2c7e1c923b</td></tr><tr><td style='border: 1px solid #dddddd'>80e6093f-5acf-4eb7-9da6-7ba9bf56a929</td><td style='border: 1px solid #dddddd'>active</td><td style='border: 1px solid #dddddd'>c015cc58-7d78-4311-8f94-0bd46cc302f8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>model_health</td><td style='border: 1px solid #dddddd'>2024-07-15 10:06:14.440000+00:00</td><td style='border: 1px solid #dddddd'>0f8e76e2-22b3-467c-b238-52f21ef2b72e</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: First 10 records were displayed.\n"
     ]
    }
   ],
   "source": [
    "wos_client.monitor_instances.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 5b6f56fb-7efc-4d85-b61c-006a80f194e0 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'5b6f56fb-7efc-4d85-b61c-006a80f194e0'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"features\": [\n",
    "        {\"feature\": \"Sex\",\n",
    "         \"majority\": ['male'],\n",
    "         \"minority\": ['female'],\n",
    "         \"threshold\": 0.95\n",
    "         },\n",
    "        {\"feature\": \"Age\",\n",
    "         \"majority\": [[26, 75]],\n",
    "         \"minority\": [[18, 25]],\n",
    "         \"threshold\": 0.95\n",
    "         }\n",
    "    ],\n",
    "    \"favourable_class\": [\"No Risk\"],\n",
    "    \"unfavourable_class\": [\"Risk\"],\n",
    "    \"min_records\": 40\n",
    "}\n",
    "\n",
    "fairness_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
    "    target=target,\n",
    "    parameters=parameters).result\n",
    "fairness_monitor_instance_id =fairness_monitor_details.metadata.id\n",
    "fairness_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Payload logging and Run fairness monitor on demand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"Results\": {\n",
      "    \"output1\": [\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"30\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_used\",\n",
      "        \"LoanAmount\": \"4864\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"34\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0314735472202301\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"6\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"vacation\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"1\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"19\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"9.4898550742073E-06\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"4\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"radio_tv\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"19\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"4.11926976084942E-06\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"39\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"5628\",\n",
      "        \"ExistingSavings\": \"500_to_1000\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"38\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0805927738547325\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"9\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"19\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"8.24751987238415E-05\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"28\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"4742\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"greater_7\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"38\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.00595719506964087\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"9\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"car_used\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"1\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"19\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"1.75070981640602E-05\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"greater_200\",\n",
      "        \"LoanDuration\": \"21\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"7148\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"43\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"Risk\",\n",
      "        \"Scored Probabilities\": \"0.9948690533638\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"17\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"car_used\",\n",
      "        \"LoanAmount\": \"3715\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"39\",\n",
      "        \"InstallmentPlans\": \"bank\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"unemployed\",\n",
      "        \"Dependents\": \"2\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.00114261277485639\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"19\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"radio_tv\",\n",
      "        \"LoanAmount\": \"2339\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"39\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0307862963527441\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"27\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"1634\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"45\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"free\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.093395009636879\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"greater_200\",\n",
      "        \"LoanDuration\": \"17\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"2083\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"1\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"33\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"free\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0781897306442261\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"24\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"car_used\",\n",
      "        \"LoanAmount\": \"2349\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"co-applicant\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"33\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0376448594033718\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"4\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"27\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"9.29525776882656E-06\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"13\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"3116\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"unemployed\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"21\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.000257308274740353\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"23\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"other\",\n",
      "        \"LoanAmount\": \"4929\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"unknown\",\n",
      "        \"Age\": \"38\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"management_self-employed\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.11523200571537\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"43\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"appliances\",\n",
      "        \"LoanAmount\": \"3661\",\n",
      "        \"ExistingSavings\": \"greater_1000\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"co-applicant\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"28\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"free\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"Risk\",\n",
      "        \"Scored Probabilities\": \"0.99767142534256\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"20\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"radio_tv\",\n",
      "        \"LoanAmount\": \"3966\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"43\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"management_self-employed\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0183224026113749\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"23\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"car_used\",\n",
      "        \"LoanAmount\": \"6913\",\n",
      "        \"ExistingSavings\": \"500_to_1000\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"co-applicant\",\n",
      "        \"CurrentResidenceDuration\": \"5\",\n",
      "        \"OwnsProperty\": \"unknown\",\n",
      "        \"Age\": \"30\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"unskilled\",\n",
      "        \"Dependents\": \"2\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"Risk\",\n",
      "        \"Scored Probabilities\": \"0.997933685779572\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"14\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"3531\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"43\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0845745950937271\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"7\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"19\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.000256493163760751\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"greater_200\",\n",
      "        \"LoanDuration\": \"20\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"1931\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"co-applicant\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"43\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.00803424511104822\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"4\",\n",
      "        \"CreditHistory\": \"no_credits\",\n",
      "        \"LoanPurpose\": \"car_used\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"30\",\n",
      "        \"InstallmentPlans\": \"bank\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"unskilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"5.99314432747633E-07\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"11\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"1546\",\n",
      "        \"ExistingSavings\": \"500_to_1000\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"28\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0920045003294945\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"18\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"greater_1000\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"19\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"unskilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.00504486355930567\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"14\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"appliances\",\n",
      "        \"LoanAmount\": \"1431\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"unemployed\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"1\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"25\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"5.46749179193284E-05\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"21\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_used\",\n",
      "        \"LoanAmount\": \"3614\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"35\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"management_self-employed\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"Risk\",\n",
      "        \"Scored Probabilities\": \"0.691476881504059\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"25\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"5330\",\n",
      "        \"ExistingSavings\": \"500_to_1000\",\n",
      "        \"EmploymentDuration\": \"greater_7\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"unknown\",\n",
      "        \"Age\": \"50\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"free\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"Risk\",\n",
      "        \"Scored Probabilities\": \"0.999527275562286\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"33\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"radio_tv\",\n",
      "        \"LoanAmount\": \"5774\",\n",
      "        \"ExistingSavings\": \"500_to_1000\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"5\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"47\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"unskilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.000324810913298279\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"4\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"250\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"unemployed\",\n",
      "        \"InstallmentPercent\": \"1\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"19\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"6.31638567938353E-06\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"28\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"car_new\",\n",
      "        \"LoanAmount\": \"1735\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"34\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.373755156993866\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"12\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"business\",\n",
      "        \"LoanAmount\": \"3223\",\n",
      "        \"ExistingSavings\": \"unknown\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"35\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"rent\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"management_self-employed\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.000141764365253039\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"17\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"radio_tv\",\n",
      "        \"LoanAmount\": \"2533\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"less_1\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"32\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.00514620635658503\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"33\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"retraining\",\n",
      "        \"LoanAmount\": \"7064\",\n",
      "        \"ExistingSavings\": \"greater_1000\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"5\",\n",
      "        \"OwnsProperty\": \"unknown\",\n",
      "        \"Age\": \"44\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"management_self-employed\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.040872722864151\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"10\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"2014\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"co-applicant\",\n",
      "        \"CurrentResidenceDuration\": \"1\",\n",
      "        \"OwnsProperty\": \"real_estate\",\n",
      "        \"Age\": \"41\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"1.92038369277725E-05\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"9\",\n",
      "        \"CreditHistory\": \"prior_payments_delayed\",\n",
      "        \"LoanPurpose\": \"radio_tv\",\n",
      "        \"LoanAmount\": \"6203\",\n",
      "        \"ExistingSavings\": \"100_to_500\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"39\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"unskilled\",\n",
      "        \"Dependents\": \"2\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.00218345248140395\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"14\",\n",
      "        \"CreditHistory\": \"all_credits_paid_back\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"1786\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"1\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"40\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0010181610705331\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"no_checking\",\n",
      "        \"LoanDuration\": \"28\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"appliances\",\n",
      "        \"LoanAmount\": \"4868\",\n",
      "        \"ExistingSavings\": \"greater_1000\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"4\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"co-applicant\",\n",
      "        \"CurrentResidenceDuration\": \"4\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"40\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"Risk\",\n",
      "        \"Scored Probabilities\": \"0.949995160102844\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"less_0\",\n",
      "        \"LoanDuration\": \"28\",\n",
      "        \"CreditHistory\": \"outstanding_credit\",\n",
      "        \"LoanPurpose\": \"appliances\",\n",
      "        \"LoanAmount\": \"2277\",\n",
      "        \"ExistingSavings\": \"greater_1000\",\n",
      "        \"EmploymentDuration\": \"4_to_7\",\n",
      "        \"InstallmentPercent\": \"2\",\n",
      "        \"Sex\": \"male\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"3\",\n",
      "        \"OwnsProperty\": \"savings_insurance\",\n",
      "        \"Age\": \"30\",\n",
      "        \"InstallmentPlans\": \"stores\",\n",
      "        \"Housing\": \"own\",\n",
      "        \"ExistingCreditsCount\": \"2\",\n",
      "        \"Job\": \"unskilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"none\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.0138161219656467\"\n",
      "      },\n",
      "      {\n",
      "        \"CheckingStatus\": \"0_to_200\",\n",
      "        \"LoanDuration\": \"20\",\n",
      "        \"CreditHistory\": \"credits_paid_to_date\",\n",
      "        \"LoanPurpose\": \"furniture\",\n",
      "        \"LoanAmount\": \"4640\",\n",
      "        \"ExistingSavings\": \"less_100\",\n",
      "        \"EmploymentDuration\": \"1_to_4\",\n",
      "        \"InstallmentPercent\": \"3\",\n",
      "        \"Sex\": \"female\",\n",
      "        \"OthersOnLoan\": \"none\",\n",
      "        \"CurrentResidenceDuration\": \"2\",\n",
      "        \"OwnsProperty\": \"car_other\",\n",
      "        \"Age\": \"39\",\n",
      "        \"InstallmentPlans\": \"none\",\n",
      "        \"Housing\": \"free\",\n",
      "        \"ExistingCreditsCount\": \"1\",\n",
      "        \"Job\": \"skilled\",\n",
      "        \"Dependents\": \"1\",\n",
      "        \"Telephone\": \"yes\",\n",
      "        \"ForeignWorker\": \"yes\",\n",
      "        \"Scored Labels\": \"No Risk\",\n",
      "        \"Scored Probabilities\": \"0.081816278398037\"\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import time\n",
    "import json\n",
    "\n",
    "scoring_url = subscription_details['entity']['deployment']['scoring_endpoint']['url']\n",
    "\n",
    "payload_data = {\n",
    "            \"Inputs\": {\n",
    "                \"input1\":\n",
    "                    training_data_pd.sample(40).drop(['Risk'],axis=1).to_dict('records'),\n",
    "            },\n",
    "            \"GlobalParameters\": {\n",
    "            }\n",
    "        }\n",
    "\n",
    "pl_body = str.encode(json.dumps(payload_data))\n",
    "\n",
    "\n",
    "token = model_asset_details['entity']['scoring_endpoint']['credentials']['token']\n",
    "headers = subscription_details['entity']['deployment']['scoring_endpoint']['request_headers']\n",
    "headers['Authorization'] = ('Bearer ' + token)\n",
    "\n",
    "start_time = time.time()\n",
    "response = requests.post(url=scoring_url, data=pl_body, headers=headers)\n",
    "response_time = int(time.time() - start_time)*1000\n",
    "result = response.json()\n",
    "\n",
    "print(json.dumps(result, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store the request and response in payload logging table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "openscale_paylod_input=convert_user_input_2_openscale(payload_data)\n",
    "openscale_payload_output=convert_user_output_2_openscale(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performing explicit payload logging.....\n",
      "Number of records in the payload logging table: 42\n"
     ]
    }
   ],
   "source": [
    "import uuid\n",
    "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
    "\n",
    "print(\"Performing explicit payload logging.....\")\n",
    "wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(\n",
    "           scoring_id=str(uuid.uuid4()),\n",
    "           request=openscale_paylod_input,\n",
    "           response=openscale_payload_output,\n",
    "           response_time=460\n",
    ")])\n",
    "time.sleep(5)\n",
    "pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\n",
    "print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data Set b5f897f0-20ec-46cb-9289-9e92c188603f Records</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_revision</th><th style='border: 1px solid #dddddd'>LoanAmount</th><th style='border: 1px solid #dddddd'>scoring_id</th><th style='border: 1px solid #dddddd'>InstallmentPercent</th><th style='border: 1px solid #dddddd'>ExistingSavings</th><th style='border: 1px solid #dddddd'>scoring_timestamp</th><th style='border: 1px solid #dddddd'>prediction_probability</th><th style='border: 1px solid #dddddd'>ProbabilityVector</th><th style='border: 1px solid #dddddd'>CheckingStatus</th><th style='border: 1px solid #dddddd'>LoanPurpose</th><th style='border: 1px solid #dddddd'>Scored Probabilities</th><th style='border: 1px solid #dddddd'>Job</th><th style='border: 1px solid #dddddd'>InstallmentPlans</th><th style='border: 1px solid #dddddd'>OthersOnLoan</th><th style='border: 1px solid #dddddd'>LoanDuration</th><th style='border: 1px solid #dddddd'>Age</th><th style='border: 1px solid #dddddd'>CreditHistory</th><th style='border: 1px solid #dddddd'>CurrentResidenceDuration</th><th style='border: 1px solid #dddddd'>ForeignWorker</th><th style='border: 1px solid #dddddd'>Housing</th><th style='border: 1px solid #dddddd'>ExistingCreditsCount</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>Telephone</th><th style='border: 1px solid #dddddd'>Scored Labels</th><th style='border: 1px solid #dddddd'>Sex</th><th style='border: 1px solid #dddddd'>Dependents</th><th style='border: 1px solid #dddddd'>OwnsProperty</th><th style='border: 1px solid #dddddd'>EmploymentDuration</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>4864</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-1</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9685264527797699</td><td style='border: 1px solid #dddddd'>[0.9685264527797699, 0.0314735472202301]</td><td style='border: 1px solid #dddddd'>less_0</td><td style='border: 1px solid #dddddd'>car_used</td><td style='border: 1px solid #dddddd'>0.0314735472202301</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>30</td><td style='border: 1px solid #dddddd'>34</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>2339</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-10</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9692137036472559</td><td style='border: 1px solid #dddddd'>[0.9692137036472559, 0.0307862963527441]</td><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>radio_tv</td><td style='border: 1px solid #dddddd'>0.0307862963527441</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>19</td><td style='border: 1px solid #dddddd'>39</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>less_1</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>1634</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-11</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.906604990363121</td><td style='border: 1px solid #dddddd'>[0.906604990363121, 0.093395009636879]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>0.093395009636879</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>27</td><td style='border: 1px solid #dddddd'>45</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>2083</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-12</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9218102693557739</td><td style='border: 1px solid #dddddd'>[0.9218102693557739, 0.0781897306442261]</td><td style='border: 1px solid #dddddd'>greater_200</td><td style='border: 1px solid #dddddd'>furniture</td><td style='border: 1px solid #dddddd'>0.0781897306442261</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>17</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>2349</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-13</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9623551405966282</td><td style='border: 1px solid #dddddd'>[0.9623551405966282, 0.0376448594033718]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>car_used</td><td style='border: 1px solid #dddddd'>0.0376448594033718</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>co-applicant</td><td style='border: 1px solid #dddddd'>24</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>250</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-14</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9999907047422312</td><td style='border: 1px solid #dddddd'>[0.9999907047422312, 9.29525776882656e-06]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>9.29525776882656e-06</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>27</td><td style='border: 1px solid #dddddd'>all_credits_paid_back</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>real_estate</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>3116</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-15</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9997426917252596</td><td style='border: 1px solid #dddddd'>[0.9997426917252596, 0.000257308274740353]</td><td style='border: 1px solid #dddddd'>less_0</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>0.000257308274740353</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>13</td><td style='border: 1px solid #dddddd'>21</td><td style='border: 1px solid #dddddd'>all_credits_paid_back</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>unemployed</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>4929</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-16</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.88476799428463</td><td style='border: 1px solid #dddddd'>[0.88476799428463, 0.11523200571537]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>other</td><td style='border: 1px solid #dddddd'>0.11523200571537</td><td style='border: 1px solid #dddddd'>management_self-employed</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>23</td><td style='border: 1px solid #dddddd'>38</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>unknown</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>3661</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-17</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>greater_1000</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.99767142534256</td><td style='border: 1px solid #dddddd'>[0.00232857465744, 0.99767142534256]</td><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>appliances</td><td style='border: 1px solid #dddddd'>0.99767142534256</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>co-applicant</td><td style='border: 1px solid #dddddd'>43</td><td style='border: 1px solid #dddddd'>28</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>3966</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-18</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9816775973886251</td><td style='border: 1px solid #dddddd'>[0.9816775973886251, 0.0183224026113749]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>radio_tv</td><td style='border: 1px solid #dddddd'>0.0183224026113749</td><td style='border: 1px solid #dddddd'>management_self-employed</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>20</td><td style='border: 1px solid #dddddd'>43</td><td style='border: 1px solid #dddddd'>prior_payments_delayed</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.data_sets.show_records(payload_data_set_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run f158fcf3-b2a4-45d5-96de-9c21bd1f25f6 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "error\n",
      "\n",
      "-------------------------------\n",
      " Run failed with status: error \n",
      "-------------------------------\n",
      "\n",
      "\n",
      "Reason: ['code: AIQMM0002E, message: Action `Trigger Monitor Run` has failed with status code 500', 'code: AIQFM5012, message: Internal error occurred during fairness computation for subscription: 1ae2f751-e220-48a9-b049-d2404311f314. Please contact support']\n"
     ]
    }
   ],
   "source": [
    "run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>5b6f56fb-7efc-4d85-b61c-006a80f194e0 Monitor Runs Metrics from: 2024-07-08 16:52:34.221207  till: 2024-07-15 16:52:34.221261</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                \n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explainability configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation 3982dccd-bb64-430f-b5e6-a4fbfb469464 \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "preparing.\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"enabled\": True\n",
    "}\n",
    "explainability_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3982dccd-bb64-430f-b5e6-a4fbfb469464'"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "explainability_monitor_id = explainability_details.metadata.id\n",
    "explainability_monitor_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run explanation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>Data Set b5f897f0-20ec-46cb-9289-9e92c188603f Records</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>asset_revision</th><th style='border: 1px solid #dddddd'>LoanAmount</th><th style='border: 1px solid #dddddd'>scoring_id</th><th style='border: 1px solid #dddddd'>InstallmentPercent</th><th style='border: 1px solid #dddddd'>ExistingSavings</th><th style='border: 1px solid #dddddd'>scoring_timestamp</th><th style='border: 1px solid #dddddd'>prediction_probability</th><th style='border: 1px solid #dddddd'>ProbabilityVector</th><th style='border: 1px solid #dddddd'>CheckingStatus</th><th style='border: 1px solid #dddddd'>LoanPurpose</th><th style='border: 1px solid #dddddd'>Scored Probabilities</th><th style='border: 1px solid #dddddd'>Job</th><th style='border: 1px solid #dddddd'>InstallmentPlans</th><th style='border: 1px solid #dddddd'>OthersOnLoan</th><th style='border: 1px solid #dddddd'>LoanDuration</th><th style='border: 1px solid #dddddd'>Age</th><th style='border: 1px solid #dddddd'>CreditHistory</th><th style='border: 1px solid #dddddd'>CurrentResidenceDuration</th><th style='border: 1px solid #dddddd'>ForeignWorker</th><th style='border: 1px solid #dddddd'>Housing</th><th style='border: 1px solid #dddddd'>ExistingCreditsCount</th><th style='border: 1px solid #dddddd'>deployment_id</th><th style='border: 1px solid #dddddd'>Telephone</th><th style='border: 1px solid #dddddd'>Scored Labels</th><th style='border: 1px solid #dddddd'>Sex</th><th style='border: 1px solid #dddddd'>Dependents</th><th style='border: 1px solid #dddddd'>OwnsProperty</th><th style='border: 1px solid #dddddd'>EmploymentDuration</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>4864</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-1</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9685264527797699</td><td style='border: 1px solid #dddddd'>[0.9685264527797699, 0.0314735472202301]</td><td style='border: 1px solid #dddddd'>less_0</td><td style='border: 1px solid #dddddd'>car_used</td><td style='border: 1px solid #dddddd'>0.0314735472202301</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>30</td><td style='border: 1px solid #dddddd'>34</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>2339</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-10</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9692137036472559</td><td style='border: 1px solid #dddddd'>[0.9692137036472559, 0.0307862963527441]</td><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>radio_tv</td><td style='border: 1px solid #dddddd'>0.0307862963527441</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>19</td><td style='border: 1px solid #dddddd'>39</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>less_1</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>1634</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-11</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.906604990363121</td><td style='border: 1px solid #dddddd'>[0.906604990363121, 0.093395009636879]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>0.093395009636879</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>27</td><td style='border: 1px solid #dddddd'>45</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>2083</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-12</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9218102693557739</td><td style='border: 1px solid #dddddd'>[0.9218102693557739, 0.0781897306442261]</td><td style='border: 1px solid #dddddd'>greater_200</td><td style='border: 1px solid #dddddd'>furniture</td><td style='border: 1px solid #dddddd'>0.0781897306442261</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>17</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>credits_paid_to_date</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>2349</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-13</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9623551405966282</td><td style='border: 1px solid #dddddd'>[0.9623551405966282, 0.0376448594033718]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>car_used</td><td style='border: 1px solid #dddddd'>0.0376448594033718</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>co-applicant</td><td style='border: 1px solid #dddddd'>24</td><td style='border: 1px solid #dddddd'>33</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>250</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-14</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9999907047422312</td><td style='border: 1px solid #dddddd'>[0.9999907047422312, 9.29525776882656e-06]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>9.29525776882656e-06</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>27</td><td style='border: 1px solid #dddddd'>all_credits_paid_back</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>real_estate</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>3116</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-15</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>less_100</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9997426917252596</td><td style='border: 1px solid #dddddd'>[0.9997426917252596, 0.000257308274740353]</td><td style='border: 1px solid #dddddd'>less_0</td><td style='border: 1px solid #dddddd'>car_new</td><td style='border: 1px solid #dddddd'>0.000257308274740353</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>13</td><td style='border: 1px solid #dddddd'>21</td><td style='border: 1px solid #dddddd'>all_credits_paid_back</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>rent</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>female</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>unemployed</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>4929</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-16</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.88476799428463</td><td style='border: 1px solid #dddddd'>[0.88476799428463, 0.11523200571537]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>other</td><td style='border: 1px solid #dddddd'>0.11523200571537</td><td style='border: 1px solid #dddddd'>management_self-employed</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>23</td><td style='border: 1px solid #dddddd'>38</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>unknown</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>3661</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-17</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>greater_1000</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.99767142534256</td><td style='border: 1px solid #dddddd'>[0.00232857465744, 0.99767142534256]</td><td style='border: 1px solid #dddddd'>no_checking</td><td style='border: 1px solid #dddddd'>appliances</td><td style='border: 1px solid #dddddd'>0.99767142534256</td><td style='border: 1px solid #dddddd'>skilled</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>co-applicant</td><td style='border: 1px solid #dddddd'>43</td><td style='border: 1px solid #dddddd'>28</td><td style='border: 1px solid #dddddd'>outstanding_credit</td><td style='border: 1px solid #dddddd'>3</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>free</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>car_other</td><td style='border: 1px solid #dddddd'>4_to_7</td></tr><tr><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>3966</td><td style='border: 1px solid #dddddd'>3211247f-cbc6-4427-b676-e9217c0bddcb-18</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>100_to_500</td><td style='border: 1px solid #dddddd'>2024-07-15T11:22:09.236Z</td><td style='border: 1px solid #dddddd'>0.9816775973886251</td><td style='border: 1px solid #dddddd'>[0.9816775973886251, 0.0183224026113749]</td><td style='border: 1px solid #dddddd'>0_to_200</td><td style='border: 1px solid #dddddd'>radio_tv</td><td style='border: 1px solid #dddddd'>0.0183224026113749</td><td style='border: 1px solid #dddddd'>management_self-employed</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>none</td><td style='border: 1px solid #dddddd'>20</td><td style='border: 1px solid #dddddd'>43</td><td style='border: 1px solid #dddddd'>prior_payments_delayed</td><td style='border: 1px solid #dddddd'>4</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>own</td><td style='border: 1px solid #dddddd'>2</td><td style='border: 1px solid #dddddd'>563f01d37f720857b95c557dc76176ad</td><td style='border: 1px solid #dddddd'>yes</td><td style='border: 1px solid #dddddd'>No Risk</td><td style='border: 1px solid #dddddd'>male</td><td style='border: 1px solid #dddddd'>1</td><td style='border: 1px solid #dddddd'>savings_insurance</td><td style='border: 1px solid #dddddd'>1_to_4</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wos_client.data_sets.show_records(data_set_id=payload_data_set_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running explanations on scoring IDs: ['3211247f-cbc6-4427-b676-e9217c0bddcb-1']\n",
      "{\n",
      "  \"metadata\": {\n",
      "    \"explanation_task_ids\": [\n",
      "      \"2872f467-52f4-4d7f-bccf-b1d247daf7db\"\n",
      "    ],\n",
      "    \"created_by\": \"IBMid-662005298W\",\n",
      "    \"created_at\": \"2024-07-15T11:22:56.121345Z\"\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2872f467-52f4-4d7f-bccf-b1d247daf7db'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pl_records_resp = wos_client.data_sets.get_list_of_records(data_set_id=payload_data_set_id, limit=1, offset=0).result\n",
    "scoring_ids = [pl_records_resp[\"records\"][0][\"entity\"][\"values\"][\"scoring_id\"]]\n",
    "print(\"Running explanations on scoring IDs: {}\".format(scoring_ids))\n",
    "explanation_types = [\"lime\", \"contrastive\"]\n",
    "result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types, subscription_id=subscription_id).result\n",
    "print(result)\n",
    "explanation_task_id=result.to_dict()['metadata']['explanation_task_ids'][0]\n",
    "explanation_task_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'explanation_task_id': '2872f467-52f4-4d7f-bccf-b1d247daf7db',\n",
       "  'created_by': 'IBMid-662005298W',\n",
       "  'created_at': '2024-07-15T11:22:56.121345Z'},\n",
       " 'entity': {'status': {'state': 'in_progress'},\n",
       "  'asset': {'id': '085460ef94636166aea5800e9ea26168',\n",
       "   'name': 'GermanCreditRisk.2019.1.9.10.41.58.611',\n",
       "   'input_data_type': 'structured',\n",
       "   'problem_type': 'binary',\n",
       "   'deployment': {'id': '563f01d37f720857b95c557dc76176ad',\n",
       "    'name': 'GermanCreditRisk.2019.1.9.10.41.58.611'}},\n",
       "  'scoring_id': '3211247f-cbc6-4427-b676-e9217c0bddcb-1'}}"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wos_client.monitor_instances.get_explanation_tasks(explanation_task_id=explanation_task_id, subscription_id=subscription_id).result.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drift monitor configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Drift requires a trained model to be uploaded manually for Azure. You can train, create and download a drift detection model using template given ( check for Drift detection model generation) [here](https://github.com/IBM-Watson/aios-data-distribution/blob/master/training_statistics_notebook.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-07-15 16:59:05--  https://github.com/IBM/watson-openscale-samples/blob/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz?raw=true\n",
      "Resolving github.com (github.com)... 20.207.73.82\n",
      "Connecting to github.com (github.com)|20.207.73.82|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://github.com/IBM/watson-openscale-samples/raw/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz [following]\n",
      "--2024-07-15 16:59:05--  https://github.com/IBM/watson-openscale-samples/raw/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz\n",
      "Reusing existing connection to github.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz [following]\n",
      "--2024-07-15 16:59:06--  https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8001::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8003::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 229416 (224K) [application/octet-stream]\n",
      "Saving to: ‘creditrisk_azure_drift_detection_model.tar.gz’\n",
      "\n",
      "creditrisk_azure_dr 100%[===================>] 224.04K   522KB/s    in 0.4s    \n",
      "\n",
      "2024-07-15 16:59:06 (522 KB/s) - ‘creditrisk_azure_drift_detection_model.tar.gz’ saved [229416/229416]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -rf creditrisk_azure_drift_detection_model.tar.gz\n",
    "!wget -O creditrisk_azure_drift_detection_model.tar.gz \"https://github.com/IBM/watson-openscale-samples/blob/main/IBM%20Cloud/Azure/assets/models/credit_risk/azure_creditrisk_drift_detection_model.tar.gz?raw=true\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<ibm_cloud_sdk_core.detailed_response.DetailedResponse at 0x137e9b520>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wos_client.monitor_instances.upload_drift_model(\n",
    "        model_path='creditrisk_azure_drift_detection_model.tar.gz',\n",
    "        data_mart_id=data_mart_id,\n",
    "        subscription_id=subscription_id\n",
    "     )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted existing drift monitor instance with id:  1eb6a66a-ea75-4aa2-9b94-2694f32c20f5\n"
     ]
    }
   ],
   "source": [
    "monitor_instances = wos_client.monitor_instances.list().result.monitor_instances\n",
    "for monitor_instance in monitor_instances:\n",
    "    monitor_def_id=monitor_instance.entity.monitor_definition_id\n",
    "    if monitor_def_id == \"drift\" and monitor_instance.entity.target.target_id == subscription_id:\n",
    "        wos_client.monitor_instances.delete(monitor_instance.metadata.id)\n",
    "        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "===================================================================================\n",
      "\n",
      " Waiting for end of monitor instance creation dc52c272-b69c-47c3-abb6-c0bb4e1cfe4d \n",
      "\n",
      "===================================================================================\n",
      "\n",
      "\n",
      "\n",
      "active\n",
      "\n",
      "---------------------------------------\n",
      " Monitor instance successfully created \n",
      "---------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'dc52c272-b69c-47c3-abb6-c0bb4e1cfe4d'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"min_samples\": 40,\n",
    "    \"drift_threshold\": 0.1,\n",
    "    \"train_drift_model\": False,\n",
    "    \"enable_model_drift\": True,\n",
    "    \"enable_data_drift\": True\n",
    "}\n",
    "\n",
    "drift_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "drift_monitor_instance_id = drift_monitor_details.metadata.id\n",
    "drift_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Drift monitor on demand"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "========================================================================\n",
      "\n",
      " Waiting for end of monitoring run a1c7ee55-ad5d-40e4-80b9-47161da367d8 \n",
      "\n",
      "========================================================================\n",
      "\n",
      "\n",
      "\n",
      "finished\n",
      "\n",
      "---------------------------\n",
      " Successfully finished run \n",
      "---------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "drift_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<HTML>\n",
       "        <body>\n",
       "            <h3>dc52c272-b69c-47c3-abb6-c0bb4e1cfe4d Monitor Runs Metrics from: 2024-07-08 17:02:10.052575  till: 2024-07-15 17:02:10.052651</h3>\n",
       "            <table style='border: 1px solid #dddddd; font-family: Courier'>\n",
       "                <th style='border: 1px solid #dddddd'>ts</th><th style='border: 1px solid #dddddd'>id</th><th style='border: 1px solid #dddddd'>measurement_id</th><th style='border: 1px solid #dddddd'>value</th><th style='border: 1px solid #dddddd'>lower_limit</th><th style='border: 1px solid #dddddd'>upper_limit</th><th style='border: 1px solid #dddddd'>tags</th><th style='border: 1px solid #dddddd'>monitor_definition_id</th><th style='border: 1px solid #dddddd'>monitor_instance_id</th><th style='border: 1px solid #dddddd'>run_id</th><th style='border: 1px solid #dddddd'>target_type</th><th style='border: 1px solid #dddddd'>target_id</th>\n",
       "                <tr><td style='border: 1px solid #dddddd'>2024-07-15 11:31:57.517922+00:00</td><td style='border: 1px solid #dddddd'>data_drift_magnitude</td><td style='border: 1px solid #dddddd'>06730537-4019-4637-b1e4-7040ffd9db58</td><td style='border: 1px solid #dddddd'>0.42857142857142855</td><td style='border: 1px solid #dddddd'>None</td><td style='border: 1px solid #dddddd'>0.1</td><td style='border: 1px solid #dddddd'>[]</td><td style='border: 1px solid #dddddd'>drift</td><td style='border: 1px solid #dddddd'>dc52c272-b69c-47c3-abb6-c0bb4e1cfe4d</td><td style='border: 1px solid #dddddd'>a1c7ee55-ad5d-40e4-80b9-47161da367d8</td><td style='border: 1px solid #dddddd'>subscription</td><td style='border: 1px solid #dddddd'>1ae2f751-e220-48a9-b049-d2404311f314</td></tr>\n",
       "            </table>\n",
       "        </body>\n",
       "        </HTML>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "time.sleep(5)\n",
    "wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have finished the tutorial for IBM Watson OpenScale and Azure Machine Learning Studio. You can now view the [OpenScale Dashboard](https://aiopenscale.cloud.ibm.com/). Click on the tile for the German Credit model to see fairness, accuracy, and performance monitors. Click on the timeseries graph to get detailed information on transactions during a specific time window.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}