Automated decision-making systems can potentially introduce biases, raising ethical concerns. This has led to the development of numerous bias mitigation techniques. However, the selection of a fairness-aware model for a specific dataset often involves a process of trial and error, as it is not always feasible to predict in advance whether the mitigation measures provided by the model will meet the user's requirements, or what impact these measures will have on other model metrics such as accuracy and run time.
Existing fairness toolkits lack a comprehensive benchmarking framework. To bridge this gap, we present FairnessEval, a framework specifically designed to evaluate fairness in Machine Learning models. FairnessEval streamlines dataset preparation, fairness evaluation, and result presentation, while also offering customization options. In this demonstration, we highlight the functionality of FairnessEval in the selection and validation of fairness-aware models. We compare various approaches and simulate deployment scenarios to showcase FairnessEval effectiveness.
Here you can find a demo notebook with working examples.
You can interact with the notebook and run the library with your experiments.
Here you can find a quick start guide to the fairnesseval API with working examples.
Parameter | Description |
---|---|
experiment_id |
ID of the experiment to run. Required. |
dataset_name |
List of dataset names. Required. |
model_name |
List of model names. Required. |
results_path RESULTS_PATH |
Path to save results. |
train_fractions |
List of fractions to be used for training. |
random_seeds |
List of random seeds to use. All random seeds set are related to this random seed. For each random_seed a new train_test split is done. |
metrics |
Metric set to be used for evaluation. Available metric set names are default , conversion_to_binary_sensitive_attribute . To use custom metrics add a new key to metrics_code_map in fairnesseval.metrics.py . |
preprocessing |
Preprocessing function to be used. Available preprocessing functions are conversion_to_binary_sensitive_attribute , binary_split_by_mean_y , default . To add a new preprocessing function add a new key to preprocessing_function_map in fairnesseval.utils_prepare_data.py . |
split_strategy |
Splitting strategy. Available split strategies are stratified_train_test_split , StratifiedKFold . |
train_test_fold |
List of train_test_fold to run with k-fold. |
model_params |
Dict with key, value pairs of model hyper parameter names (key) and list of values to be iterated (values). When multiple list of parameters are specified the cross product is used to generate all the combinations to test. |
debug |
Debug mode if set, the program will stop at the first exception. |
This table provides a clear and concise overview of the parameters and their descriptions.