training_statistics_notebook_for_3_5_10.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Notebook for generating training data distribution and configuring Fairness\n",
    "\n",
    "This notebook analyzes training data and outputs a JSON which contains information related to data distribution and fairness configuration.  In order to use this notebook you need to do the following:\n",
    "\n",
    "1. Read the training data into a pandas dataframe called \"data_df\".  There is sample code below to show how this can be done if the training data is in IBM Cloud Object Storage. \n",
    "2. Edit the below cells and provide the training data and fairness configuration information. \n",
    "3. Run the notebook. It will generate a JSON and a download link for the JSON will be present at the very end of the notebook.\n",
    "4. Download the JSON by clicking on the link and upload it in the IBM AI OpenScale GUI.\n",
    "\n",
    "If you have multiple models (deployments), you will have to repeat the above steps for each model (deployment).\n",
    "\n",
    "**Note:** Please restart the kernel after executing below cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas numpy pyspark ibm-cos-sdk | tail -n 1\n",
    "!pip install --upgrade ibm-watson-openscale | tail -n 1\n",
    "!pip install ibm-wos-utils==2.1.5 | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read training data into a pandas data frame\n",
    "\n",
    "The first thing that you need to do is to read the training data into a pandas dataframe called \"data_df\".  Given below is sample code for doing this if the training data is in IBM Cloud Object Storage.  Please edit the below cell and make changes so that you can read your training data from the location where it is stored.  Please ensure that the training data is present in a data frame called \"data_df\".\n",
    "\n",
    "*Note: Pandas' read\\_csv method converts the columns to its data types. If you want the column type to not be interpreted, specify the dtype param to read_csv method in this cell. More on this method [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)*\n",
    "\n",
    "*Note: By default NA values will be dropped while computing training data distribution and training the drift archive. Please ensure to handle the NA values during Pandas' read\\_csv method*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ----------------------------------------------------------------------------------------------------\n",
    "# IBM Confidential\n",
    "# OCO Source Materials\n",
    "# 5900-A3Q, 5737-H76\n",
    "# Copyright IBM Corp. 2018, 2021\n",
    "# The source code for this Notebook is not published or other-wise divested of its trade \n",
    "# secrets, irrespective of what has been deposited with the U.S.Copyright Office.\n",
    "# ----------------------------------------------------------------------------------------------------\n",
    "\n",
    "VERSION = 5.0\n",
    "\n",
    "# code to read file in COS to pandas dataframe object\n",
    "import sys\n",
    "import types\n",
    "import pandas as pd\n",
    "from ibm_botocore.client import Config\n",
    "import ibm_boto3\n",
    "\n",
    "def __iter__(self): return 0\n",
    "\n",
    "api_key = \"<API Key>\"\n",
    "resource_instance_id = \"crn:v1:bluemix:public:cloud-object-storage:global:a/111111aaa1a111aa11d111111aa11111:22b22bbb-b22b-22bb-2b22-22b22bB22b2b::\"\n",
    "auth_endpoint = \"https://iam.ng.bluemix.net/oidc/token\"\n",
    "service_endpoint =  \"https://s3-api.dal-us-geo.objectstorage.softlayer.net\"\n",
    "bucket =  \"<Bucket Name>\"\n",
    "file_name= \"<File Name>\"\n",
    "\n",
    "cos_client = ibm_boto3.client(service_name=\"s3\",\n",
    "    ibm_api_key_id=api_key,\n",
    "    ibm_auth_endpoint=auth_endpoint,\n",
    "    config=Config(signature_version=\"oauth\"),\n",
    "    endpoint_url=service_endpoint)\n",
    "\n",
    "body = cos_client.get_object(Bucket=bucket,Key=file_name)[\"Body\"]\n",
    "\n",
    "# add missing __iter__ method, so pandas accepts body as file-like object\n",
    "if not hasattr(body, \"__iter__\"): body.__iter__ = types.MethodType( __iter__, body )\n",
    "\n",
    "data_df = pd.read_csv(body)\n",
    "data_df.head()\n",
    "\n",
    "#Print columns from data frams\n",
    "#print(\"column names:{}\".format(list(data_df.columns.values)))\n",
    "\n",
    "# Uncomment following 2 lines if you want to read training data from local CSV file when running through local Jupyter notebook\n",
    "#data_df = pd.read_csv(\"<FULLPATH_TO_CSV_FILE>\")\n",
    "#data_df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Select the services for which configuration information needs to be generated\n",
    "\n",
    "This notebook has support to generaton configuration information related to fairness , explainability and drift service. The below can be used by the user to control service specific configuration information.\n",
    "\n",
    "Details of the service speicifc flags available:\n",
    "\n",
    "- enable_fairness : Flag to allow generation of fairness specific data distribution needed for configuration\n",
    "- enable_explainability : Flag to allow generation of explainability specific information\n",
    "- enable_drift: Flag to allow generation of drift detection model needed by drift service\n",
    "\n",
    "```\n",
    "service_configuration_support = {\n",
    "        \"enable_fairness\": True,\n",
    "        \"enable_explainability\": True,    \n",
    "        \"enable_drift\": True  \n",
    "    }  \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_configuration_support = {\n",
    "    \"enable_fairness\": True,\n",
    "    \"enable_explainability\": True,\n",
    "    \"enable_drift\": True\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Data and Fairness Configuration Information\n",
    "\n",
    "Please provide information about the training data which is used to train the model.  In order to explain the configuration better, let us first consider an example of a Loan Processing Model which is trying to predict whether a person should get a loan or not. The training data for such a model will potentially contain the following columns: Credit_History, Monthly_salary, Applicant_Age, Loan_amount, Gender, Marital_status, Approval.  The \"Approval\" column contains the target field (label column or class label) and it can have the following values: \"Loan Granted\", \"Loan Denied\" or \"Loan Partially Granted\".  In this model we would like to ensure that the model is not biased against Gender=Female or Gender=Transgender.  We would also like to ensure that the model is not biased against the age group 15 to 30 years or age group 61 to 120 years. \n",
    "\n",
    "For the above model, the configuration information that we need to provide is:\n",
    "\n",
    "- class_label:  This is the name of the column in the training data dataframe (data_df) which contains the target field (also known as label column or the class label).  For the Loan Processing Model it would be \"Approval\".\n",
    "- feature_columns: This is a comma separated list of column names which contain the feature column names (in the training data dataframe data_df).  For the Loan Processing model this would be: [\"Credit_History\", \"Monthly_salary\", \"Applicant_Age\", \"Loan_amount\", \"Gender\", \"Marital_status\"]\n",
    "- categorical_columns: The list of column names (in data_df) which contain categorical values.  This should also include those columns which originally contained categorical values and have now been converted to numeric values. E.g., in the Loan Processing Model, the Marital_status column originally could have values: Single, Married, Divorced, Separated, Widowed.  These could have been converted to numeric values as follows: Single -> 0, Married -> 1, Divorced -> 2, Separated -> 3 and Widowed -> 4.  Thus the training data will have numeric values.  Please identify such columns as categorical.  Thus the list of categorical columns for the Loan Processing Model will be Credit_History, Gender and Marital_status. \n",
    "\n",
    "For the Loan Processing Model, this information will be provided as follows:\n",
    "\n",
    "training_data_info = { <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"class_label\": \"Approval\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"feature_columns\": [\"Credit_History\", \"Monthly_salary\", \"Applicant_Age\", \"Loan_amount\", \"Gender\", \"Marital_status\"],    \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;\"categorical_columns\": [\"Credit_History\",\"Gender\",\"Marital_status\"]   \n",
    "    }  \n",
    "    \n",
    "  **Note:** Please note that categorical columns selected should be subset of feature columns. If there are no categorical columns among the feature columns selected , please set \"categorical_columns as [] or None\"\n",
    "\n",
    "Please edit the next cell and provide the above information for your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data_info = {\n",
    "    \"class_label\": \"<EDIT THIS>\",\n",
    "    \"feature_columns\": [\"<EDIT THIS>\"],\n",
    "    \"categorical_columns\": [\"<EDIT THIS>\"]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify the Model Type\n",
    "\n",
    "In the next cell, specify the type of your model.  If your model is a binary classification model, then set the type to \"binary\". If it is a multi-class classifier then set the type to \"multiclass\". If it is a regression model (e.g., Linear Regression), then set it to \"regression\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Set model_type. Acceptable values are:[\"binary\",\"multiclass\",\"regression\"]\n",
    "model_type = \"binary\"\n",
    "#model_type = \"multiclass\"\n",
    "#model_type = \"regression\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify the Fairness Configuration\n",
    "\n",
    "You need to provide the following information for the fairness configuration: \n",
    "\n",
    "- fairness_attributes:  These are the attributes on which you wish to monitor fairness. In the Loan Processing Model, we wanted to ensure that the model is not baised against people of specific age group and people belonging to a specific gender.  Hence \"Applicant_Age\" and \"Gender\" will be the fairness attributes for the Loan Processing Model.\n",
    "- With Indirect Bias support, you can also monitor protected attributes for fairness. The protected attributes are those attributes which are present in the training data but are not used to train the model. For example, sensitive attributes like gender, race, age may be present in training data but are not used for training. To check if there exists indirect bias with respect to some protected attribute due to possible correlation with some feature column, it can be specified in fairness configuration.\n",
    "- type: The data type of the fairness attribute (e.g., float or int or double)\n",
    "- minority:  The minority group for which we want to ensure that the model is not biased.  For the Loan Processing Model we wanted to ensure that the model is not biased against people in the age group 15 to 30 years & 61 to 120 years as well as people with Gender = Female or Gender = Transgender.  Hence the minority group for the fairness attribute \"Applicant_Age\" will be [15,30] and [61,120] and the minority group for fairness attribute \"Gender\" will be: \"Female\", \"Transgender\".  \n",
    "- majority: The majority group for which the model might be biased towards.  For the Loan Processing Model, the majority group for the fairness attribute \"Applicant_Age\" will be [31,60], i.e., all the ages except the minority group.  For the fairness attribute \"Gender\" the majority group will be: \"Male\".  \n",
    "- threshold:  The fairness threshold beyond which the Model is considered to be biased.  For the Loan Processing Model, let us say that the Bank is willing to tolerate the fact that Female and Transgender applicants will get upto 20% lesser approved loans than Males.  However, if the percentage is more than 20% then the Loan Processing Model will be considered biased.  E.g., if the percentage of approved loans for Female or Transgender applicants is say 25% lesser than those approved for Male applicants then the Model is to be considered as acting in a biased manner.  Thus for this scenario, the Fairness threshold will be 80 (100-20) (this is represented as a value normalized to 1, i.e., 0.8).  \n",
    "\n",
    "The fairness attributes for Loan Processing Model will be specified as:\n",
    "\n",
    "fairness_attributes = [  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"feature\": \"Applicant_Age\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"type\" : \"int\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"majority\": [ [31,60] ],   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"minority\": [ [15, 30], [61,120] ],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"threshold\" : 0.8  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;},  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"feature\": \"Gender\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"type\" : \"string\",   \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"majority\": [\"Male\"],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"minority\": [\"Female\", \"Transgender\"],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"threshold\" : 0.8  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;]  \n",
    "\n",
    "Please edit the next cell and provide the fairness configuration for your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fairness_attributes = [{\n",
    "                           \"type\" : \"<DATA_TYPE>\", #data type of the column eg: float or int or double\n",
    "                           \"feature\": \"<COLUMN_NAME>\", \n",
    "                           \"majority\": [\n",
    "                               [X, Y] # range of values for column eg: [31, 45] for int or [31.4, 45.1] for float\n",
    "                           ],\n",
    "                           \"minority\": [\n",
    "                               [A, B], # range of values for column eg: [10, 15] for int or [10.5, 15.5] for float\n",
    "                               [C, D]   # range of values for column eg: [80, 100] for int or [80.0, 99.9] for float                    \n",
    "                           ],\n",
    "                           \"threshold\": <VALUE> #such that 0<VALUE<=1. eg: 0.8\n",
    "                       }]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify the Favorable and Unfavorable class values\n",
    "\n",
    "The second part of fairness configuration is about the favourable and unfavourable class values.  Recall that in the case of Loan Processing Model, the target field (label column or class label) can have the following values: \"Loan Granted\", \"Loan Denied\" and \"Loan Partially Granted\".  Out of these values \"Loan Granted\" and \"Loan Partially Granted\" can be considered as being favorable and \"Loan Denied\" is unfavorable.  In other words in order to measure fairness, we need to know the target field values which can be considered as being favourable and those values which can be considered as unfavourable.  \n",
    "\n",
    "For the Loan Prediction Model, the values can be specified as follows:\n",
    "\n",
    "parameters = {  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"favourable_class\" :  [ \"Loan Granted\", \"Loan Partially Granted\" ],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"unfavourable_class\": [ \"Loan Denied\" ]  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;}  \n",
    "\n",
    "In case of a regression models, the favourable and unfavourable classes will be ranges.  For example, for a model which predicts medicine dosage, the favorable outcome could be between 80 ml to 120 ml or between 5 ml to 20 ml whereas unfavorable outcome will be values between 21 ml to 79ml.  For such a model, the favorable and unfavorable values will be specified as follows:\n",
    "     \n",
    "parameters = {  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"favourable_class\" :  [ [5, 20], [80, 120] ],  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\"unfavourable_class\": [ [21, 79] ]  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;}  \n",
    "\n",
    "Please edit the next cell to provide information about your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For classification models use the below.\n",
    "parameters = {\n",
    "        \"favourable_class\" :  [ \"<EDIT THIS>\", \"<EDIT THIS>\" ],\n",
    "        \"unfavourable_class\": [ \"<EDIT THIS>\" ]\n",
    "    }\n",
    "# For regression models use the below.  Delete the entry which is not required.\n",
    "parameters = {\n",
    "        \"favourable_class\" :  [ [<EDIT THIS>, <EDIT THIS>], [<EDIT THIS>,<EDIT THIS>] ],\n",
    "        \"unfavourable_class\": [ [<EDIT THIS>, <EDIT THIS>] ]\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify the number of records which should be processed for Fairness\n",
    "\n",
    "The final piece of information that needs to be provided is the number of records (min_records) that should be used for computing the fairness. Fairness checks runs hourly.  If min_records is set to 5000, then every hour fairness checking will pick up the last 5000 records which were sent to the model for scoring and compute the fairness on those 5000 records.  Please note that fairness computation will not start till the time that 5000 records are sent to the model for scoring.\n",
    "\n",
    "If we set the value of \"min_records\" to a small number, then fairness computation will get influenced by the scoring requests sent to the model in the recent past. In other words, the model might be flagged as being biased if it is acting in a biased manner on the last few records, but overall it might not be acting in a biased manner.  On the other hand, if the \"min_records\" is set to a very large number, then we will not be able to catch model bias quickly. Hence the value of min_records should be set such that it is neither too small or too large.\n",
    "\n",
    "Please updated the next cell to specify a value for min_records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# min_records = <Minimum number of records to be considered for preforming scoring>\n",
    "min_records = <EDIT THIS>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Input \n",
    "\n",
    "You need not edit anything beyond this point.  Run the notebook and go to the very last cell.  There will be a link to download the JSON file (called: \"Download training data distribution JSON file\").  Download the file and upload it using the IBM AI OpenScale GUI.\n",
    "\n",
    "*Note: drop_na parameter of TrainingStats object should be set to \"False\" if NA values are taken care while reading the training data in the above cells*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_openscale.utils.training_stats import TrainingStats\n",
    "\n",
    "enable_explainability = service_configuration_support.get(\"enable_explainability\")\n",
    "enable_fairness = service_configuration_support.get(\"enable_fairness\")\n",
    "\n",
    "if enable_explainability or enable_fairness:\n",
    "    fairness_inputs = None\n",
    "    if enable_fairness:\n",
    "        fairness_inputs = {\n",
    "                \"fairness_attributes\": fairness_attributes,\n",
    "                \"min_records\" : min_records,\n",
    "                \"favourable_class\" :  parameters[\"favourable_class\"],\n",
    "                \"unfavourable_class\": parameters[\"unfavourable_class\"]\n",
    "            }\n",
    "    \n",
    "    input_parameters = {\n",
    "        \"label_column\": training_data_info[\"class_label\"],\n",
    "        \"feature_columns\": training_data_info[\"feature_columns\"],\n",
    "        \"categorical_columns\": training_data_info[\"categorical_columns\"],\n",
    "        \"fairness_inputs\": fairness_inputs,  \n",
    "        \"problem_type\" : model_type  \n",
    "    }\n",
    "\n",
    "    training_stats = TrainingStats(data_df,input_parameters, explain=enable_explainability, fairness=enable_fairness, drop_na=True)\n",
    "    config_json = training_stats.get_training_statistics()\n",
    "    config_json[\"notebook_version\"] = VERSION\n",
    "#print(config_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indirect Bias\n",
    "In case of Indirect bias i.e if protected attributes(the sensitive attributes like race, gender etc which are present in the training data but are not used to train the model) are being monitored for fairness:\n",
    "- Bias service identifies correlations between the protected attribute and model features. Correlated attributes are also known as proxy features.\n",
    "- Existence of correlations with model features can result in indirect bias w.r.t protected attribute even though it is not used to train the model.\n",
    "- Highly correlated attributes based on their correlation strength are considered while computing bias for a given protected attribute.\n",
    "\n",
    "The following cell identifies if user has configured protected attribute for fairness by checking the feature, non-feature columns and the fairness configuration. If protected attribute/s are configured then it identifies correlations and stores it in the fairness configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Checking if protected attributes are configured for fairness monitoring. If yes, then computing correlation information for each meta-field and updating it in the fairness configuration\n",
    "if enable_fairness:\n",
    "    fairness_configuration = config_json.get(\"fairness_configuration\")\n",
    "    training_columns = data_df.columns.tolist()\n",
    "    label_column = training_data_info.get(\"class_label\")\n",
    "    training_columns.remove(label_column)\n",
    "    feature_columns = training_data_info.get(\"feature_columns\")\n",
    "    non_feature_columns = list(set(training_columns) - set(feature_columns))\n",
    "    if non_feature_columns is not None and len(non_feature_columns) > 0:\n",
    "        protected_attributes = []\n",
    "        fairness_attributes_list = [attribute.get(\"feature\") for attribute in fairness_attributes]\n",
    "        for col in non_feature_columns:\n",
    "            if col in fairness_attributes_list:\n",
    "                protected_attributes.append(col)\n",
    "        if len(protected_attributes) > 0:\n",
    "            from ibm_watson_openscale.utils.indirect_bias_processor import IndirectBiasProcessor\n",
    "            fairness_configuration = IndirectBiasProcessor().get_correlated_attributes(data_df, fairness_configuration, feature_columns, protected_attributes, label_column)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "print(\"Finished generating training distribution data\")\n",
    "\n",
    "# Create a file download link\n",
    "import base64\n",
    "from IPython.display import HTML\n",
    "\n",
    "def create_download_link( title = \"Download training data distribution JSON file\", filename = \"training_distribution.json\"):  \n",
    "    if enable_explainability or enable_fairness:\n",
    "        output_json = json.dumps(config_json, indent=2)\n",
    "        b64 = base64.b64encode(output_json.encode())\n",
    "        payload = b64.decode()\n",
    "        html = '<a download=\"{filename}\" href=\"data:text/json;base64,{payload}\" target=\"_blank\">{title}</a>'\n",
    "        html = html.format(payload=payload,title=title,filename=filename)\n",
    "        return HTML(html)\n",
    "    else:\n",
    "        print(\"No download link generated as fairness/explainability services are disabled.\")\n",
    "\n",
    "create_download_link()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Drift detection model generation\n",
    "\n",
    "Please update the score function which will be used forgenerating drift detection model which will used for drift detection . This might take sometime to generate model and time taken depends on the training dataset size. The output of the score function should be a 2 arrays 1. Array of model prediction 2. Array of probabilities \n",
    "\n",
    "- User is expected to make sure that the data type of the \"class label\" column selected and the prediction column are same . For eg : If class label is numeric , the prediction array should also be numeric\n",
    "\n",
    "- Each entry of a probability array should have all the probabities of the unique class lable .\n",
    "  For eg: If the model_type=multiclass and unique class labels are A, B, C, D . Each entry in the probability array should be a array of size 4 . Eg : [ [0.50,0.30,0.10,0.10] ,[0.40,0.20,0.30,0.10]...]\n",
    "  \n",
    "**Note:**\n",
    "- *User is expected to add \"score\" method , which should output prediction column array and probability column array.*\n",
    "- *The data type of the label column and prediction column should be same . User needs to make sure that label column and prediction column array should have the same unique class labels*\n",
    "- **Please update the score function below with the help of templates documented [here](https://github.com/IBM-Watson/aios-data-distribution/blob/master/Score%20function%20templates%20for%20drift%20detection.md)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Update score function\n",
    "# def score(training_data_frame){\n",
    "#     <Fill in the template using the score function templates provided>\n",
    "# }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Generate drift detection model\n",
    "from ibm_wos_utils.drift.drift_trainer import DriftTrainer\n",
    "enable_drift = service_configuration_support.get(\"enable_drift\")\n",
    "if enable_drift:\n",
    "    drift_detection_input = {\n",
    "        \"feature_columns\":training_data_info.get(\"feature_columns\"),\n",
    "        \"categorical_columns\":training_data_info.get(\"categorical_columns\"),\n",
    "        \"label_column\": training_data_info.get(\"class_label\"),\n",
    "        \"problem_type\": model_type\n",
    "    }\n",
    "    \n",
    "    drift_trainer = DriftTrainer(data_df,drift_detection_input)\n",
    "    if model_type != \"regression\":\n",
    "        #Note: batch_size can be customized by user as per the training data size\n",
    "        drift_trainer.generate_drift_detection_model(score,batch_size=data_df.shape[0])\n",
    "    \n",
    "    #Note:\n",
    "    # - Two column constraints are not computed beyond two_column_learner_limit(default set to 200)\n",
    "    # - Categorical columns with large (determined by categorical_unique_threshold; default > 0.8) number of unique values relative to total rows in the column are discarded. \n",
    "    #User can adjust the value depending on the requirement\n",
    "    \n",
    "    drift_trainer.learn_constraints(two_column_learner_limit=200, categorical_unique_threshold=0.8)\n",
    "    drift_trainer.create_archive()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Generate a download link for drift detection model\n",
    "from IPython.display import HTML\n",
    "import base64\n",
    "import io\n",
    "\n",
    "def create_download_link_for_ddm( title = \"Download Drift detection model\", filename = \"drift_detection_model.tar.gz\"):  \n",
    "    \n",
    "    #Retains stats information    \n",
    "    if enable_drift:\n",
    "        with open(filename,\"rb\") as file:\n",
    "            ddm = file.read()\n",
    "        b64 = base64.b64encode(ddm)\n",
    "        payload = b64.decode()\n",
    "        \n",
    "        html = '<a download=\"{filename}\" href=\"data:text/json;base64,{payload}\" target=\"_blank\">{title}</a>'\n",
    "        html = html.format(payload=payload,title=title,filename=filename)\n",
    "        return HTML(html)\n",
    "    else:\n",
    "        print(\"Drift Detection is not enabled. Please enable and rerun the notebook\")\n",
    "\n",
    "create_download_link_for_ddm()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}