diff --git a/.github/workflows/azureml-cpu-nightly.yml b/.github/workflows/azureml-cpu-nightly.yml
index 34869f2c14..0f7d54041b 100644
--- a/.github/workflows/azureml-cpu-nightly.yml
+++ b/.github/workflows/azureml-cpu-nightly.yml
@@ -14,7 +14,7 @@ on:
# │ │ │ │ │
# │ │ │ │ │
schedule:
- - cron: '0 0 */2 * *' # basically running every other day at 12AM
+ - cron: '0 0 */5 * *' # running every 5 days at 12AM
# cron works with default branch (main) only: # https://github.community/t/on-schedule-per-branch/17525/2
push:
diff --git a/.github/workflows/azureml-gpu-nightly.yml b/.github/workflows/azureml-gpu-nightly.yml
index 0e39eb1dfe..df4ab17109 100644
--- a/.github/workflows/azureml-gpu-nightly.yml
+++ b/.github/workflows/azureml-gpu-nightly.yml
@@ -14,7 +14,7 @@ on:
# │ │ │ │ │
# │ │ │ │ │
schedule:
- - cron: '0 0 */2 * *' # basically running every other day at 12AM
+ - cron: '0 0 */5 * *' # running every 5 days at 12AM
# cron works with default branch (main) only: # https://github.community/t/on-schedule-per-branch/17525/2
push:
diff --git a/.github/workflows/azureml-spark-nightly.yml b/.github/workflows/azureml-spark-nightly.yml
index 99675b5993..7069731224 100644
--- a/.github/workflows/azureml-spark-nightly.yml
+++ b/.github/workflows/azureml-spark-nightly.yml
@@ -14,7 +14,7 @@ on:
# │ │ │ │ │
# │ │ │ │ │
schedule:
- - cron: '0 0 */2 * *' # basically running every other day at 12AM
+ - cron: '0 0 */5 * *' # running every 5 days at 12AM
# cron works with default branch (main) only: # https://github.community/t/on-schedule-per-branch/17525/2
push:
diff --git a/examples/03_evaluate/evaluation.ipynb b/examples/03_evaluate/evaluation.ipynb
index 7f8feb6b59..e67b0691c6 100644
--- a/examples/03_evaluate/evaluation.ipynb
+++ b/examples/03_evaluate/evaluation.ipynb
@@ -58,7 +58,6 @@
"source": [
"# set the environment path to find Recommenders\n",
"import sys\n",
-
"import pandas as pd\n",
"import pyspark\n",
"from sklearn.preprocessing import minmax_scale\n",
@@ -387,7 +386,7 @@
"* **the recommender is to predict ranking instead of explicit rating**. For example, if the consumer of the recommender cares about the ranked recommended items, rating metrics do not apply directly. Usually a relevancy function such as top-k will be applied to generate the ranked list from predicted ratings in order to evaluate the recommender with other metrics. \n",
"* **the recommender is to generate recommendation scores that have different scales with the original ratings (e.g., the SAR algorithm)**. In this case, the difference between the generated scores and the original scores (or, ratings) is not valid for measuring accuracy of the model.\n",
"\n",
- "#### 2.1.2 How-to with the evaluation utilities\n",
+ "#### 2.1.2 How to work with the evaluation utilities\n",
"\n",
"A few notes about the interface of the Rating evaluator class:\n",
"1. The columns of user, item, and rating (prediction) should be present in the ground-truth DataFrame (prediction DataFrame).\n",
@@ -539,7 +538,7 @@
"source": [
"|Metric|Range|Selection criteria|Limitation|Reference|\n",
"|------|-------------------------------|---------|----------|---------|\n",
- "|RMSE|$> 0$|The smaller the better.|May be biased, and less explainable than MSE|[link](https://en.wikipedia.org/wiki/Root-mean-square_deviation)|\n",
+ "|RMSE|$> 0$|The smaller the better.|May be biased, and less explainable than MAE|[link](https://en.wikipedia.org/wiki/Root-mean-square_deviation)|\n",
"|R2|$\\leq 1$|The closer to $1$ the better.|Depend on variable distributions.|[link](https://en.wikipedia.org/wiki/Coefficient_of_determination)|\n",
"|MAE|$\\geq 0$|The smaller the better.|Dependent on variable scale.|[link](https://en.wikipedia.org/wiki/Mean_absolute_error)|\n",
"|Explained variance|$\\leq 1$|The closer to $1$ the better.|Depend on variable distributions.|[link](https://en.wikipedia.org/wiki/Explained_variation)|"
@@ -556,7 +555,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "\"Beyond-accuray evaluation\" was proposed to evaluate how relevant recommendations are for users. In this case, a recommendation system is a treated as a ranking system. Given a relency definition, recommendation system outputs a list of recommended items to each user, which is ordered by relevance. The evaluation part takes ground-truth data, the actual items that users interact with (e.g., liked, purchased, etc.), and the recommendation data, as inputs, to calculate ranking evaluation metrics. \n",
+ "\"Beyond-accuray evaluation\" was proposed to evaluate how relevant recommendations are for users. In this case, a recommendation system is a treated as a ranking system. Given relency definition, recommendation system outputs a list of recommended items to each user, which is ordered by relevance. The evaluation part takes ground-truth data, the actual items that users interact with (e.g., liked, purchased, etc.), and the recommendation data, as inputs, to calculate ranking evaluation metrics. \n",
"\n",
"#### 2.2.1 Use cases\n",
"\n",
@@ -576,7 +575,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "#### 2.2.1 Relevancy of recommendation\n",
+ "#### 2.2.3 Relevancy of recommendation\n",
"\n",
"Relevancy of recommendation can be measured in different ways:\n",
"\n",
@@ -641,7 +640,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "#### 2.2.1 Precision\n",
+ "#### 2.2.4 Precision\n",
"\n",
"Precision@k is a metric that evaluates how many items in the recommendation list are relevant (hit) in the ground-truth data. For each user the precision score is normalized by `k` and then the overall precision scores are averaged by the total number of users. \n",
"\n",
@@ -669,7 +668,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "#### 2.2.2 Recall\n",
+ "#### 2.2.5 Recall\n",
"\n",
"Recall@k is a metric that evaluates how many relevant items in the ground-truth data are in the recommendation list. For each user the recall score is normalized by the total number of ground-truth items and then the overall recall scores are averaged by the total number of users. "
]
@@ -695,7 +694,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "#### 2.2.3 Normalized Discounted Cumulative Gain (NDCG)\n",
+ "#### 2.2.6 Normalized Discounted Cumulative Gain (NDCG)\n",
"\n",
"NDCG is a metric that evaluates how well the recommender performs in recommending ranked items to users. Therefore both hit of relevant items and correctness in ranking of these items matter to the NDCG evaluation. The total NDCG score is normalized by the total number of users."
]
@@ -721,7 +720,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "#### 2.2.4 Mean Average Precision (MAP)\n",
+ "#### 2.2.7 Mean Average Precision (MAP)\n",
"\n",
"MAP is a metric that evaluates the average precision for each user in the datasets. It also penalizes ranking correctness of the recommended items. The overall MAP score is normalized by the total number of users."
]
@@ -747,7 +746,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "#### 2.2.5 ROC and AUC\n",
+ "#### 2.2.8 ROC and AUC\n",
"\n",
"ROC, as well as AUC, is a well known metric that is used for evaluating binary classification problem. It is similar in the case of binary rating typed recommendation algorithm where the \"hit\" accuracy on the relevant items is used for measuring the recommender's performance. \n",
"\n",
@@ -1891,7 +1890,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "#### 2.2.5 Summary"
+ "#### 2.3 Summary"
]
},
{
diff --git a/examples/06_benchmarks/README.md b/examples/06_benchmarks/README.md
index 26ac91e475..2f8320a654 100644
--- a/examples/06_benchmarks/README.md
+++ b/examples/06_benchmarks/README.md
@@ -2,8 +2,6 @@
In this folder we show benchmarks using different algorithms. To facilitate the benchmark computation, we provide a set of wrapper functions that can be found in the file [benchmark_utils.py](benchmark_utils.py).
-The machine we used to perform the benchmarks is a Standard NC6s_v2 [Azure DSVM](https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/) (6 vCPUs, 112 GB memory and 1 P100 GPU). Spark ALS is run in local standalone mode.
-
## MovieLens
[MovieLens](https://grouplens.org/datasets/movielens/) is one of the most common datasets used in the literature in Recommendation Systems. The dataset consists of a collection of users, movies and movie ratings, there are several available sizes:
@@ -13,6 +11,4 @@ The machine we used to perform the benchmarks is a Standard NC6s_v2 [Azure DSVM]
* MovieLens 10M: 10 million ratings from 72000 users on 10000 movies.
* MovieLens 20M: 20 million ratings from 138000 users on 27000 movies
-The MovieLens benchmark can be seen at [movielens.ipynb](movielens.ipynb). In this notebook, the MovieLens dataset is split into training / test sets using a stratified splitting method that takes 75% of each user's ratings as training data, and the remaining 25% ratings as test data. For ranking metrics we use `k=10` (top 10 recommended items). The algorithms used in this benchmark are [ALS](../00_quick_start/als_movielens.ipynb), [SVD](../02_model_collaborative_filtering/surprise_svd_deep_dive.ipynb), [SAR](../00_quick_start/sar_movielens.ipynb), [NCF](../00_quick_start/ncf_movielens.ipynb), [BPR](../02_model_collaborative_filtering/cornac_bpr_deep_dive.ipynb), [BiVAE](../02_model_collaborative_filtering/cornac_bivae_deep_dive.ipynb), [LightGCN](../02_model_collaborative_filtering/lightgcn_deep_dive.ipynb) and [FastAI](../00_quick_start/fastai_movielens.ipynb).
-
-
+The MovieLens benchmark can be seen at [movielens.ipynb](movielens.ipynb). This illustrative comparison applies to collaborative filtering algorithms available in this repository such as [Spark ALS](../00_quick_start/als_movielens.ipynb), [SVD](../02_model_collaborative_filtering/surprise_svd_deep_dive.ipynb), [SAR](../00_quick_start/sar_movielens.ipynb), [LightGCN](../02_model_collaborative_filtering/lightgcn_deep_dive.ipynb) and others using the Movielens dataset, using three environments (CPU, GPU and Spark). These algorithms are usable in a variety of recommendation tasks, including product or news recommendations.
diff --git a/examples/06_benchmarks/benchmark_utils.py b/examples/06_benchmarks/benchmark_utils.py
index b79dbb3b68..561eea4d40 100644
--- a/examples/06_benchmarks/benchmark_utils.py
+++ b/examples/06_benchmarks/benchmark_utils.py
@@ -1,5 +1,10 @@
-import pandas as pd
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License.
+
+import os
import numpy as np
+import pandas as pd
+from tempfile import TemporaryDirectory
from pyspark.ml.recommendation import ALS
from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import FloatType, IntegerType, LongType
@@ -47,6 +52,12 @@
from recommenders.evaluation.python_evaluation import rmse, mae, rsquared, exp_var
+# Helpers
+tmp_dir = TemporaryDirectory()
+TRAIN_FILE = os.path.join(tmp_dir.name, "df_train.csv")
+TEST_FILE = os.path.join(tmp_dir.name, "df_test.csv")
+
+
def prepare_training_als(train, test):
schema = StructType(
(
@@ -57,7 +68,7 @@ def prepare_training_als(train, test):
)
)
spark = start_or_get_spark()
- return spark.createDataFrame(train, schema)
+ return spark.createDataFrame(train, schema).cache()
def train_als(params, data):
@@ -77,7 +88,7 @@ def prepare_metrics_als(train, test):
)
)
spark = start_or_get_spark()
- return spark.createDataFrame(train, schema), spark.createDataFrame(test, schema)
+ return spark.createDataFrame(train, schema).cache(), spark.createDataFrame(test, schema).cache()
def predict_als(model, test):
@@ -223,13 +234,20 @@ def recommend_k_fastai(model, test, train, top_k=DEFAULT_K, remove_seen=True):
return topk_scores, t
-def prepare_training_ncf(train, test):
+def prepare_training_ncf(df_train, df_test):
+ #df_train.sort_values(["userID"], axis=0, ascending=[True], inplace=True)
+ #df_test.sort_values(["userID"], axis=0, ascending=[True], inplace=True)
+ train = df_train.sort_values(["userID"], axis=0, ascending=[True])
+ test = df_test.sort_values(["userID"], axis=0, ascending=[True])
+ test = test[df_test["userID"].isin(train["userID"].unique())]
+ test = test[test["itemID"].isin(train["itemID"].unique())]
+ train.to_csv(TRAIN_FILE, index=False)
+ test.to_csv(TEST_FILE, index=False)
return NCFDataset(
- train=train,
+ train_file=TRAIN_FILE,
col_user=DEFAULT_USER_COL,
col_item=DEFAULT_ITEM_COL,
col_rating=DEFAULT_RATING_COL,
- col_timestamp=DEFAULT_TIMESTAMP_COL,
seed=SEED,
)
@@ -263,6 +281,7 @@ def recommend_k_ncf(model, test, train, top_k=DEFAULT_K, remove_seen=True):
topk_scores = merged[merged[DEFAULT_RATING_COL].isnull()].drop(
DEFAULT_RATING_COL, axis=1
)
+ # Remove temp files
return topk_scores, t
diff --git a/examples/06_benchmarks/movielens.ipynb b/examples/06_benchmarks/movielens.ipynb
index 6abd934bff..28dc5ceb1c 100644
--- a/examples/06_benchmarks/movielens.ipynb
+++ b/examples/06_benchmarks/movielens.ipynb
@@ -26,7 +26,7 @@
"\n",
"* Environment\n",
" * The comparison is run on a [Azure Data Science Virtual Machine](https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/). \n",
- " * The virtual machine size is Standard NC6 (6 vcpus, 55 GB memory, 1K80 GPU).\n",
+ " * The virtual machine size is a [Standard_NC6s_v2](https://learn.microsoft.com/es-es/azure/virtual-machines/ncv2-series) with 6 CPUs, 112Gb of RAM, and 1 GPU NVIDIA Tesla P100 with 16Gb of memory.\n",
" * It should be noted that the single node DSVM is not supposed to run scalable benchmarking analysis. Either scaling up or out the computing instances is necessary to run the benchmarking in an run-time efficient way without any memory issue.\n",
" * **NOTE ABOUT THE DEPENDENCIES TO INSTALL**: This notebook uses CPU, GPU and PySpark algorithms, so make sure you install the `full environment` as detailed in the [SETUP.md](../../SETUP.md). \n",
" \n",
@@ -71,41 +71,55 @@
"cell_type": "code",
"execution_count": 1,
"metadata": {},
+ "outputs": [],
+ "source": [
+ "import warnings\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "import logging\n",
+ "logging.basicConfig(level=logging.ERROR) "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "System version: 3.6.11 | packaged by conda-forge | (default, Nov 27 2020, 18:57:37) \n",
- "[GCC 9.3.0]\n",
- "Pandas version: 1.1.5\n",
- "PySpark version: 2.4.5\n",
+ "System version: 3.7.13 (default, Mar 29 2022, 02:18:16) \n",
+ "[GCC 7.5.0]\n",
+ "NumPy version: 1.21.6\n",
+ "Pandas version: 1.3.5\n",
+ "PySpark version: 3.2.2\n",
"Surprise version: 1.1.1\n",
- "PyTorch version: 1.4.0\n",
- "Fast AI version: 1.0.46\n",
- "Cornac version: 1.12.0\n",
- "Tensorflow version: 1.15.4\n",
- "CUDA version: CUDA Version 10.2.89\n",
- "CuDNN version: No CUDNN in this machine\n",
- "Number of cores: 48\n"
+ "PyTorch version: 1.12.1+cu102\n",
+ "Fast AI version: 1.0.61\n",
+ "Cornac version: 1.14.2\n",
+ "TensorFlow version: 2.7.4\n",
+ "CUDA version: 10.2\n",
+ "CuDNN version: 7605\n",
+ "Number of cores: 6\n"
]
}
],
"source": [
- "import sys\n",
-
"import os\n",
+ "import sys\n",
"import json\n",
"import pandas as pd\n",
"import numpy as np\n",
"import seaborn as sns\n",
"import pyspark\n",
+ "import tensorflow as tf # NOTE: TF needs to be imported before PyTorch, otherwise we get a weird initialization error\n",
+ "tf.get_logger().setLevel('ERROR') # only show error messages\n",
"import torch\n",
"import fastai\n",
- "import tensorflow as tf\n",
- "tf.get_logger().setLevel('ERROR') # only show error messages\n",
"import surprise\n",
+ "import cornac\n",
"\n",
+ "from recommenders.utils.spark_utils import start_or_get_spark\n",
"from recommenders.utils.general_utils import get_number_processors\n",
"from recommenders.utils.gpu_utils import get_cuda_version, get_cudnn_version\n",
"from recommenders.datasets import movielens\n",
@@ -114,33 +128,36 @@
"\n",
"from benchmark_utils import * \n",
"\n",
- "print(\"System version: {}\".format(sys.version))\n",
- "print(\"Pandas version: {}\".format(pd.__version__))\n",
- "print(\"PySpark version: {}\".format(pyspark.__version__))\n",
- "print(\"Surprise version: {}\".format(surprise.__version__))\n",
- "print(\"PyTorch version: {}\".format(torch.__version__))\n",
- "print(\"Fast AI version: {}\".format(fastai.__version__))\n",
- "print(\"Cornac version: {}\".format(cornac.__version__))\n",
- "print(\"Tensorflow version: {}\".format(tf.__version__))\n",
- "print(\"CUDA version: {}\".format(get_cuda_version()))\n",
- "print(\"CuDNN version: {}\".format(get_cudnn_version()))\n",
- "n_cores = get_number_processors()\n",
- "print(\"Number of cores: {}\".format(n_cores))\n",
+ "print(f\"System version: {sys.version}\")\n",
+ "print(f\"NumPy version: {np.__version__}\")\n",
+ "print(f\"Pandas version: {pd.__version__}\")\n",
+ "print(f\"PySpark version: {pyspark.__version__}\")\n",
+ "print(f\"Surprise version: {surprise.__version__}\")\n",
+ "print(f\"PyTorch version: {torch.__version__}\")\n",
+ "print(f\"Fast AI version: {fastai.__version__}\")\n",
+ "print(f\"Cornac version: {cornac.__version__}\")\n",
+ "print(f\"TensorFlow version: {tf.__version__}\")\n",
+ "print(f\"CUDA version: {get_cuda_version()}\")\n",
+ "print(f\"CuDNN version: {get_cudnn_version()}\")\n",
+ "print(f\"Number of cores: {get_number_processors()}\")\n",
"\n",
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
+ "execution_count": null,
"metadata": {},
+ "outputs": [],
"source": [
- "## Parameters"
+ "spark = start_or_get_spark(\"PySpark\", memory=\"32g\")\n",
+ "spark.conf.set(\"spark.sql.analyzer.failAmbiguousSelfJoin\", \"false\")"
]
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
@@ -150,7 +167,7 @@
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
@@ -160,9 +177,30 @@
"torch.cuda.manual_seed_all(SEED)"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Parameters"
+ ]
+ },
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": null,
+ "metadata": {
+ "tags": [
+ "parameters"
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "data_sizes = [\"100k\", \"1m\"] # Movielens data size: 100k, 1m, 10m, or 20m\n",
+ "algorithms = [\"als\", \"svd\", \"sar\", \"ncf\", \"fastai\", \"bpr\", \"bivae\", \"lightgcn\"]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
@@ -198,13 +236,13 @@
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"als_params = {\n",
" \"rank\": 10,\n",
- " \"maxIter\": 15,\n",
+ " \"maxIter\": 20,\n",
" \"implicitPrefs\": False,\n",
" \"alpha\": 0.1,\n",
" \"regParam\": 0.05,\n",
@@ -276,10 +314,10 @@
"}\n",
"\n",
"lightgcn_param = {\n",
- " \"yaml_file\": os.path.join(\"..\",\"..\",\"recommenders\", \"recommender\", \"deeprec\", \"config\", \"lightgcn.yaml\"),\n",
+ " \"yaml_file\": os.path.join(\"..\",\"..\",\"recommenders\", \"models\", \"deeprec\", \"config\", \"lightgcn.yaml\"),\n",
" \"n_layers\": 3,\n",
" \"batch_size\": 1024,\n",
- " \"epochs\": 15,\n",
+ " \"epochs\": 20,\n",
" \"learning_rate\": 0.005,\n",
" \"eval_epoch\": 5,\n",
" \"top_k\": DEFAULT_K,\n",
@@ -299,7 +337,7 @@
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
@@ -317,7 +355,7 @@
},
{
"cell_type": "code",
- "execution_count": 8,
+ "execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
@@ -329,7 +367,7 @@
},
{
"cell_type": "code",
- "execution_count": 9,
+ "execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
@@ -347,7 +385,7 @@
},
{
"cell_type": "code",
- "execution_count": 10,
+ "execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
@@ -360,7 +398,7 @@
},
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
@@ -378,7 +416,7 @@
},
{
"cell_type": "code",
- "execution_count": 12,
+ "execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
@@ -403,7 +441,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
@@ -437,17 +475,7 @@
},
{
"cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [],
- "source": [
- "data_sizes = [\"100k\", \"1m\"] # Movielens data size: 100k, 1m, 10m, or 20m\n",
- "algorithms = [\"als\", \"svd\", \"sar\", \"ncf\", \"fastai\", \"bpr\", \"bivae\", \"lightgcn\"]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
+ "execution_count": 16,
"metadata": {
"scrolled": true
},
@@ -456,7 +484,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 4.81k/4.81k [00:01<00:00, 2.41kKB/s]\n"
+ "100%|██████████| 4.81k/4.81k [00:00<00:00, 12.5kKB/s]\n"
]
},
{
@@ -465,65 +493,109 @@
"text": [
"Size of Movielens 100k: (100000, 4)\n",
"\n",
- "Computing als algorithm on Movielens 100k\n",
- "Training time: 6.6739s\n",
- "Rating prediction time: 0.0716s\n",
- "Ranking prediction time: 0.0910s\n",
+ "Computing als algorithm on Movielens 100k\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " \r"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Training time: 6.8526s\n",
+ "Rating prediction time: 0.0587s\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "22/10/19 08:58:41 WARN Column: Constructing trivially true equals predicate, 'userID#225 = userID#225'. Perhaps you need to use aliases.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Ranking prediction time: 0.0782s\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " \r"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
"\n",
"Computing svd algorithm on Movielens 100k\n",
- "Training time: 4.2722s\n",
- "Rating prediction time: 0.2882s\n",
- "Ranking prediction time: 14.9461s\n",
+ "Training time: 4.0902s\n",
+ "Rating prediction time: 0.2698s\n",
+ "Ranking prediction time: 13.8704s\n",
"\n",
"Computing sar algorithm on Movielens 100k\n",
- "Training time: 0.3608s\n",
- "Ranking prediction time: 0.0853s\n",
+ "Training time: 0.3344s\n",
+ "Ranking prediction time: 0.0836s\n",
"\n",
"Computing ncf algorithm on Movielens 100k\n",
- "Training time: 63.8578s\n",
- "Ranking prediction time: 4.8731s\n",
+ "Training time: 66.8339s\n",
+ "Ranking prediction time: 3.5393s\n",
"\n",
"Computing fastai algorithm on Movielens 100k\n",
- "Training time: 96.2693s\n",
- "Rating prediction time: 0.0533s\n",
- "Ranking prediction time: 3.2034s\n",
+ "Training time: 69.7469s\n",
+ "Rating prediction time: 0.0338s\n",
+ "Ranking prediction time: 2.7415s\n",
"\n",
"Computing bpr algorithm on Movielens 100k\n",
- "Training time: 6.6029s\n",
- "Ranking prediction time: 1.5408s\n",
+ "Training time: 5.8205s\n",
+ "Ranking prediction time: 1.9365s\n",
"\n",
"Computing bivae algorithm on Movielens 100k\n",
- "Training time: 10.4133s\n",
- "Ranking prediction time: 1.5687s\n",
+ "Training time: 11.4762s\n",
+ "Ranking prediction time: 1.4382s\n",
"\n",
"Computing lightgcn algorithm on Movielens 100k\n",
"Already create adjacency matrix.\n",
"Already normalize adjacency matrix.\n",
"Using xavier initialization.\n",
- "Epoch 1 (train)1.5s: train loss = 0.47708 = (mf)0.47684 + (embed)0.00024\n",
- "Epoch 2 (train)0.9s: train loss = 0.28938 = (mf)0.28875 + (embed)0.00063\n",
- "Epoch 3 (train)0.9s: train loss = 0.25478 = (mf)0.25398 + (embed)0.00080\n",
- "Epoch 4 (train)0.9s: train loss = 0.23985 = (mf)0.23888 + (embed)0.00097\n",
- "Epoch 5 (train)0.8s + (eval)0.2s: train loss = 0.23025 = (mf)0.22914 + (embed)0.00110, recall = 0.16051, ndcg = 0.34891, precision = 0.30297, map = 0.09295\n",
- "Epoch 6 (train)0.9s: train loss = 0.22287 = (mf)0.22165 + (embed)0.00122\n",
- "Epoch 7 (train)0.9s: train loss = 0.21368 = (mf)0.21236 + (embed)0.00132\n",
- "Epoch 8 (train)0.9s: train loss = 0.20252 = (mf)0.20107 + (embed)0.00145\n",
- "Epoch 9 (train)0.9s: train loss = 0.19008 = (mf)0.18847 + (embed)0.00161\n",
- "Epoch 10 (train)0.9s + (eval)0.2s: train loss = 0.18022 = (mf)0.17844 + (embed)0.00178, recall = 0.18071, ndcg = 0.39217, precision = 0.34019, map = 0.10928\n",
- "Epoch 11 (train)0.9s: train loss = 0.17680 = (mf)0.17487 + (embed)0.00194\n",
- "Epoch 12 (train)1.0s: train loss = 0.17439 = (mf)0.17233 + (embed)0.00206\n",
- "Epoch 13 (train)0.9s: train loss = 0.16669 = (mf)0.16450 + (embed)0.00219\n",
- "Epoch 14 (train)0.9s: train loss = 0.16579 = (mf)0.16349 + (embed)0.00230\n",
- "Epoch 15 (train)0.9s + (eval)0.2s: train loss = 0.16282 = (mf)0.16041 + (embed)0.00241, recall = 0.19053, ndcg = 0.40763, precision = 0.35292, map = 0.11737\n",
- "Training time: 14.7697s\n",
- "Ranking prediction time: 0.0577s\n"
+ "Epoch 1 (train)0.9s: train loss = 0.47340 = (mf)0.47316 + (embed)0.00024\n",
+ "Epoch 2 (train)0.8s: train loss = 0.28803 = (mf)0.28739 + (embed)0.00064\n",
+ "Epoch 3 (train)0.8s: train loss = 0.25425 = (mf)0.25343 + (embed)0.00082\n",
+ "Epoch 4 (train)0.8s: train loss = 0.23797 = (mf)0.23699 + (embed)0.00098\n",
+ "Epoch 5 (train)0.8s + (eval)0.2s: train loss = 0.22717 = (mf)0.22605 + (embed)0.00111, recall = 0.16053, ndcg = 0.34968, precision = 0.30276, map = 0.09269\n",
+ "Epoch 6 (train)0.8s: train loss = 0.22202 = (mf)0.22081 + (embed)0.00121\n",
+ "Epoch 7 (train)0.8s: train loss = 0.21388 = (mf)0.21256 + (embed)0.00132\n",
+ "Epoch 8 (train)0.8s: train loss = 0.20578 = (mf)0.20434 + (embed)0.00144\n",
+ "Epoch 9 (train)0.8s: train loss = 0.19345 = (mf)0.19186 + (embed)0.00158\n",
+ "Epoch 10 (train)0.8s + (eval)0.2s: train loss = 0.18439 = (mf)0.18265 + (embed)0.00175, recall = 0.18087, ndcg = 0.39271, precision = 0.34316, map = 0.10908\n",
+ "Epoch 11 (train)0.8s: train loss = 0.17564 = (mf)0.17374 + (embed)0.00190\n",
+ "Epoch 12 (train)0.8s: train loss = 0.16834 = (mf)0.16629 + (embed)0.00205\n",
+ "Epoch 13 (train)0.8s: train loss = 0.17051 = (mf)0.16833 + (embed)0.00218\n",
+ "Epoch 14 (train)0.8s: train loss = 0.16736 = (mf)0.16508 + (embed)0.00228\n",
+ "Epoch 15 (train)0.8s + (eval)0.2s: train loss = 0.16324 = (mf)0.16086 + (embed)0.00238, recall = 0.19198, ndcg = 0.40608, precision = 0.35048, map = 0.11731\n",
+ "Epoch 16 (train)0.8s: train loss = 0.16130 = (mf)0.15882 + (embed)0.00249\n",
+ "Epoch 17 (train)0.8s: train loss = 0.15924 = (mf)0.15667 + (embed)0.00257\n",
+ "Epoch 18 (train)0.8s: train loss = 0.15797 = (mf)0.15531 + (embed)0.00266\n",
+ "Epoch 19 (train)0.8s: train loss = 0.15601 = (mf)0.15325 + (embed)0.00275\n",
+ "Epoch 20 (train)0.8s + (eval)0.2s: train loss = 0.15112 = (mf)0.14826 + (embed)0.00286, recall = 0.19605, ndcg = 0.41763, precision = 0.36098, map = 0.12163\n",
+ "Training time: 16.2290s\n",
+ "Ranking prediction time: 0.0451s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 5.78k/5.78k [00:01<00:00, 2.89kKB/s]\n"
+ "100%|██████████| 5.78k/5.78k [00:00<00:00, 15.4kKB/s]\n"
]
},
{
@@ -532,62 +604,110 @@
"text": [
"Size of Movielens 1m: (1000209, 4)\n",
"\n",
- "Computing als algorithm on Movielens 1m\n",
- "Training time: 3.3558s\n",
- "Rating prediction time: 0.0262s\n",
- "Ranking prediction time: 0.0547s\n",
+ "Computing als algorithm on Movielens 1m\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "22/10/19 09:03:02 WARN TaskSetManager: Stage 588 contains a task of very large size (2759 KiB). The maximum recommended task size is 1000 KiB.\n",
+ "22/10/19 09:03:02 WARN TaskSetManager: Stage 589 contains a task of very large size (2759 KiB). The maximum recommended task size is 1000 KiB.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Training time: 7.0365s\n",
+ "Rating prediction time: 0.0355s\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "22/10/19 09:03:19 WARN Column: Constructing trivially true equals predicate, 'userID#2403 = userID#2403'. Perhaps you need to use aliases.\n",
+ "22/10/19 09:03:19 WARN TaskSetManager: Stage 1045 contains a task of very large size (2759 KiB). The maximum recommended task size is 1000 KiB.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Ranking prediction time: 0.0491s\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "22/10/19 09:03:19 WARN TaskSetManager: Stage 1046 contains a task of very large size (2759 KiB). The maximum recommended task size is 1000 KiB.\n",
+ "22/10/19 09:03:20 WARN TaskSetManager: Stage 1092 contains a task of very large size (2759 KiB). The maximum recommended task size is 1000 KiB.\n",
+ " \r"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
"\n",
"Computing svd algorithm on Movielens 1m\n",
- "Training time: 43.4842s\n",
- "Rating prediction time: 3.4605s\n",
- "Ranking prediction time: 210.9069s\n",
+ "Training time: 41.6351s\n",
+ "Rating prediction time: 2.8386s\n",
+ "Ranking prediction time: 190.6115s\n",
"\n",
"Computing sar algorithm on Movielens 1m\n",
- "Training time: 3.3966s\n",
- "Ranking prediction time: 2.1385s\n",
+ "Training time: 3.2292s\n",
+ "Ranking prediction time: 1.9796s\n",
"\n",
"Computing ncf algorithm on Movielens 1m\n",
- "Training time: 818.8540s\n",
- "Ranking prediction time: 52.5052s\n",
+ "Training time: 816.1049s\n",
+ "Ranking prediction time: 48.9872s\n",
"\n",
"Computing fastai algorithm on Movielens 1m\n",
- "Training time: 800.6882s\n",
- "Rating prediction time: 0.4382s\n",
- "Ranking prediction time: 54.6923s\n",
+ "Training time: 663.2788s\n",
+ "Rating prediction time: 0.3985s\n",
+ "Ranking prediction time: 47.2290s\n",
"\n",
"Computing bpr algorithm on Movielens 1m\n",
- "Training time: 70.1201s\n",
- "Ranking prediction time: 30.3135s\n",
+ "Training time: 66.0371s\n",
+ "Ranking prediction time: 27.4882s\n",
"\n",
"Computing bivae algorithm on Movielens 1m\n",
- "Training time: 157.1440s\n",
- "Ranking prediction time: 29.6179s\n",
+ "Training time: 152.0912s\n",
+ "Ranking prediction time: 27.5727s\n",
"\n",
"Computing lightgcn algorithm on Movielens 1m\n",
"Already create adjacency matrix.\n",
"Already normalize adjacency matrix.\n",
"Using xavier initialization.\n",
- "Epoch 1 (train)33.9s: train loss = 0.34573 = (mf)0.34513 + (embed)0.00060\n",
- "Epoch 2 (train)27.2s: train loss = 0.26890 = (mf)0.26748 + (embed)0.00142\n",
- "Epoch 3 (train)27.2s: train loss = 0.22710 = (mf)0.22481 + (embed)0.00228\n",
- "Epoch 4 (train)27.2s: train loss = 0.20468 = (mf)0.20173 + (embed)0.00295\n",
- "Epoch 5 (train)28.2s + (eval)2.3s: train loss = 0.18749 = (mf)0.18387 + (embed)0.00362, recall = 0.12346, ndcg = 0.37514, precision = 0.33876, map = 0.07333\n",
- "Epoch 6 (train)28.0s: train loss = 0.17191 = (mf)0.16761 + (embed)0.00430\n",
- "Epoch 7 (train)27.3s: train loss = 0.16209 = (mf)0.15716 + (embed)0.00493\n",
- "Epoch 8 (train)28.0s: train loss = 0.15376 = (mf)0.14825 + (embed)0.00551\n",
- "Epoch 9 (train)27.6s: train loss = 0.14711 = (mf)0.14105 + (embed)0.00606\n",
- "Epoch 10 (train)25.7s + (eval)2.0s: train loss = 0.14080 = (mf)0.13424 + (embed)0.00657, recall = 0.13965, ndcg = 0.40881, precision = 0.37071, map = 0.08435\n",
- "Epoch 11 (train)25.5s: train loss = 0.13659 = (mf)0.12955 + (embed)0.00704\n",
- "Epoch 12 (train)26.1s: train loss = 0.13223 = (mf)0.12474 + (embed)0.00749\n",
- "Epoch 13 (train)26.4s: train loss = 0.12833 = (mf)0.12039 + (embed)0.00794\n",
- "Epoch 14 (train)25.7s: train loss = 0.12402 = (mf)0.11564 + (embed)0.00838\n",
- "Epoch 15 (train)27.1s + (eval)2.0s: train loss = 0.12167 = (mf)0.11285 + (embed)0.00882, recall = 0.14387, ndcg = 0.41690, precision = 0.37853, map = 0.08785\n",
- "Training time: 417.1986s\n",
- "Ranking prediction time: 0.5851s\n",
+ "Epoch 1 (train)23.2s: train loss = 0.34771 = (mf)0.34712 + (embed)0.00059\n",
+ "Epoch 2 (train)22.8s: train loss = 0.27739 = (mf)0.27605 + (embed)0.00134\n",
+ "Epoch 3 (train)22.8s: train loss = 0.22916 = (mf)0.22690 + (embed)0.00226\n",
+ "Epoch 4 (train)22.8s: train loss = 0.20559 = (mf)0.20265 + (embed)0.00295\n",
+ "Epoch 5 (train)22.8s + (eval)1.9s: train loss = 0.18818 = (mf)0.18458 + (embed)0.00360, recall = 0.12101, ndcg = 0.36960, precision = 0.33409, map = 0.07156\n",
+ "Epoch 6 (train)22.8s: train loss = 0.17528 = (mf)0.17106 + (embed)0.00422\n",
+ "Epoch 7 (train)22.8s: train loss = 0.16460 = (mf)0.15977 + (embed)0.00482\n",
+ "Epoch 8 (train)22.8s: train loss = 0.15525 = (mf)0.14984 + (embed)0.00541\n",
+ "Epoch 9 (train)22.8s: train loss = 0.14808 = (mf)0.14214 + (embed)0.00595\n",
+ "Epoch 10 (train)22.8s + (eval)1.8s: train loss = 0.14349 = (mf)0.13703 + (embed)0.00647, recall = 0.13872, ndcg = 0.40819, precision = 0.37030, map = 0.08408\n",
+ "Epoch 11 (train)22.8s: train loss = 0.13819 = (mf)0.13122 + (embed)0.00696\n",
+ "Epoch 12 (train)22.8s: train loss = 0.13232 = (mf)0.12490 + (embed)0.00742\n",
+ "Epoch 13 (train)22.8s: train loss = 0.12975 = (mf)0.12188 + (embed)0.00787\n",
+ "Epoch 14 (train)22.8s: train loss = 0.12524 = (mf)0.11694 + (embed)0.00830\n",
+ "Epoch 15 (train)22.8s + (eval)1.8s: train loss = 0.12257 = (mf)0.11385 + (embed)0.00872, recall = 0.14381, ndcg = 0.41610, precision = 0.37805, map = 0.08683\n",
+ "Epoch 16 (train)22.8s: train loss = 0.11986 = (mf)0.11074 + (embed)0.00912\n",
+ "Epoch 17 (train)22.7s: train loss = 0.11695 = (mf)0.10744 + (embed)0.00952\n",
+ "Epoch 18 (train)23.0s: train loss = 0.11496 = (mf)0.10506 + (embed)0.00990\n",
+ "Epoch 19 (train)22.8s: train loss = 0.11245 = (mf)0.10217 + (embed)0.01028\n",
+ "Epoch 20 (train)22.8s + (eval)1.8s: train loss = 0.11068 = (mf)0.10004 + (embed)0.01063, recall = 0.14773, ndcg = 0.42390, precision = 0.38572, map = 0.08977\n",
+ "Training time: 463.8591s\n",
+ "Ranking prediction time: 0.5867s\n",
"\n",
"Computation finished\n",
- "CPU times: user 58min 40s, sys: 8min 2s, total: 1h 6min 43s\n",
- "Wall time: 1h 3min 3s\n"
+ "CPU times: user 55min 50s, sys: 8min 57s, total: 1h 4min 47s\n",
+ "Wall time: 56min 59s\n"
]
}
],
@@ -658,7 +778,7 @@
" summary = generate_summary(data_size, algo, DEFAULT_K, time_train, time_rating, ratings, time_ranking, rankings)\n",
" df_results.loc[df_results.shape[0] + 1] = summary\n",
" \n",
- "print(\"\\nComputation finished\")"
+ "print(\"\\nComputation finished\")\n"
]
},
{
@@ -670,7 +790,7 @@
},
{
"cell_type": "code",
- "execution_count": 16,
+ "execution_count": 17,
"metadata": {},
"outputs": [
{
@@ -716,30 +836,30 @@
" | 100k | \n",
" als | \n",
" 10 | \n",
- " 6.6739 | \n",
- " 0.0716 | \n",
- " 0.963173 | \n",
- " 0.752521 | \n",
- " 0.258453 | \n",
- " 0.254536 | \n",
- " 0.0910 | \n",
- " 0.004091 | \n",
- " 0.041346 | \n",
- " 0.045387 | \n",
- " 0.015172 | \n",
+ " 6.8526 | \n",
+ " 0.0587 | \n",
+ " 0.962863 | \n",
+ " 0.747088 | \n",
+ " 0.258405 | \n",
+ " 0.255016 | \n",
+ " 0.0782 | \n",
+ " 0.004697 | \n",
+ " 0.046619 | \n",
+ " 0.049629 | \n",
+ " 0.016688 | \n",
" \n",
" \n",
" | 2 | \n",
" 100k | \n",
" svd | \n",
" 10 | \n",
- " 4.2722 | \n",
- " 0.2882 | \n",
+ " 4.0902 | \n",
+ " 0.2698 | \n",
" 0.938681 | \n",
" 0.742690 | \n",
" 0.291967 | \n",
" 0.291971 | \n",
- " 14.9461 | \n",
+ " 13.8704 | \n",
" 0.012873 | \n",
" 0.095930 | \n",
" 0.091198 | \n",
@@ -750,13 +870,13 @@
" 100k | \n",
" sar | \n",
" 10 | \n",
- " 0.3608 | \n",
+ " 0.3344 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 0.0853 | \n",
+ " 0.0836 | \n",
" 0.113028 | \n",
" 0.388321 | \n",
" 0.333828 | \n",
@@ -767,47 +887,47 @@
" 100k | \n",
" ncf | \n",
" 10 | \n",
- " 63.8578 | \n",
+ " 66.8339 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 4.8731 | \n",
- " 0.094349 | \n",
- " 0.369862 | \n",
- " 0.330117 | \n",
- " 0.165953 | \n",
+ " 3.5393 | \n",
+ " 0.108609 | \n",
+ " 0.398754 | \n",
+ " 0.349735 | \n",
+ " 0.181576 | \n",
"
\n",
" \n",
" | 5 | \n",
" 100k | \n",
" fastai | \n",
" 10 | \n",
- " 96.2693 | \n",
- " 0.0533 | \n",
- " 0.943093 | \n",
- " 0.744332 | \n",
- " 0.285296 | \n",
- " 0.287658 | \n",
- " 3.2034 | \n",
- " 0.025590 | \n",
- " 0.148119 | \n",
- " 0.130541 | \n",
- " 0.053903 | \n",
+ " 69.7469 | \n",
+ " 0.0338 | \n",
+ " 0.942754 | \n",
+ " 0.745138 | \n",
+ " 0.285810 | \n",
+ " 0.288468 | \n",
+ " 2.7415 | \n",
+ " 0.025896 | \n",
+ " 0.151481 | \n",
+ " 0.131813 | \n",
+ " 0.054491 | \n",
"
\n",
" \n",
" | 6 | \n",
" 100k | \n",
" bpr | \n",
" 10 | \n",
- " 6.6029 | \n",
+ " 5.8205 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 1.5408 | \n",
+ " 1.9365 | \n",
" 0.132478 | \n",
" 0.441997 | \n",
" 0.388229 | \n",
@@ -818,13 +938,13 @@
" 100k | \n",
" bivae | \n",
" 10 | \n",
- " 10.4133 | \n",
+ " 11.4762 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 1.5687 | \n",
+ " 1.4382 | \n",
" 0.146126 | \n",
" 0.475077 | \n",
" 0.411771 | \n",
@@ -835,47 +955,47 @@
" 100k | \n",
" lightgcn | \n",
" 10 | \n",
- " 14.7697 | \n",
+ " 16.2290 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 0.0577 | \n",
- " 0.117371 | \n",
- " 0.407634 | \n",
- " 0.352916 | \n",
- " 0.190530 | \n",
+ " 0.0451 | \n",
+ " 0.121633 | \n",
+ " 0.417629 | \n",
+ " 0.360976 | \n",
+ " 0.196052 | \n",
"
\n",
" \n",
" | 9 | \n",
" 1m | \n",
" als | \n",
" 10 | \n",
- " 3.3558 | \n",
- " 0.0262 | \n",
- " 0.860522 | \n",
- " 0.679683 | \n",
- " 0.412199 | \n",
- " 0.406352 | \n",
- " 0.0547 | \n",
- " 0.002039 | \n",
- " 0.025166 | \n",
- " 0.031373 | \n",
- " 0.009830 | \n",
+ " 7.0365 | \n",
+ " 0.0355 | \n",
+ " 0.858791 | \n",
+ " 0.677568 | \n",
+ " 0.413262 | \n",
+ " 0.408737 | \n",
+ " 0.0491 | \n",
+ " 0.002683 | \n",
+ " 0.030447 | \n",
+ " 0.036707 | \n",
+ " 0.011461 | \n",
"
\n",
" \n",
" | 10 | \n",
" 1m | \n",
" svd | \n",
" 10 | \n",
- " 43.4842 | \n",
- " 3.4605 | \n",
+ " 41.6351 | \n",
+ " 2.8386 | \n",
" 0.883017 | \n",
" 0.695366 | \n",
" 0.374910 | \n",
" 0.374911 | \n",
- " 210.9069 | \n",
+ " 190.6115 | \n",
" 0.008828 | \n",
" 0.089320 | \n",
" 0.082856 | \n",
@@ -886,13 +1006,13 @@
" 1m | \n",
" sar | \n",
" 10 | \n",
- " 3.3966 | \n",
+ " 3.2292 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 2.1385 | \n",
+ " 1.9796 | \n",
" 0.066214 | \n",
" 0.313502 | \n",
" 0.279692 | \n",
@@ -903,47 +1023,47 @@
" 1m | \n",
" ncf | \n",
" 10 | \n",
- " 818.8540 | \n",
+ " 816.1049 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 52.5052 | \n",
- " 0.063974 | \n",
- " 0.348130 | \n",
- " 0.318900 | \n",
- " 0.109476 | \n",
+ " 48.9872 | \n",
+ " 0.065931 | \n",
+ " 0.357964 | \n",
+ " 0.327249 | \n",
+ " 0.111665 | \n",
"
\n",
" \n",
" | 13 | \n",
" 1m | \n",
" fastai | \n",
" 10 | \n",
- " 800.6882 | \n",
- " 0.4382 | \n",
- " 0.874729 | \n",
- " 0.695722 | \n",
- " 0.386589 | \n",
- " 0.389092 | \n",
- " 54.6923 | \n",
- " 0.026299 | \n",
- " 0.183996 | \n",
- " 0.167550 | \n",
- " 0.055682 | \n",
+ " 663.2788 | \n",
+ " 0.3985 | \n",
+ " 0.874907 | \n",
+ " 0.695758 | \n",
+ " 0.386338 | \n",
+ " 0.388787 | \n",
+ " 47.2290 | \n",
+ " 0.026237 | \n",
+ " 0.184787 | \n",
+ " 0.168494 | \n",
+ " 0.056014 | \n",
"
\n",
" \n",
" | 14 | \n",
" 1m | \n",
" bpr | \n",
" 10 | \n",
- " 70.1201 | \n",
+ " 66.0371 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 30.3135 | \n",
+ " 27.4882 | \n",
" 0.083546 | \n",
" 0.390911 | \n",
" 0.357777 | \n",
@@ -954,34 +1074,34 @@
" 1m | \n",
" bivae | \n",
" 10 | \n",
- " 157.1440 | \n",
+ " 152.0912 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 29.6179 | \n",
- " 0.092682 | \n",
- " 0.433709 | \n",
- " 0.396058 | \n",
- " 0.148564 | \n",
+ " 27.5727 | \n",
+ " 0.094632 | \n",
+ " 0.438106 | \n",
+ " 0.399039 | \n",
+ " 0.149889 | \n",
"
\n",
" \n",
" | 16 | \n",
" 1m | \n",
" lightgcn | \n",
" 10 | \n",
- " 417.1986 | \n",
+ " 463.8591 | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
- " 0.5851 | \n",
- " 0.087846 | \n",
- " 0.416899 | \n",
- " 0.378532 | \n",
- " 0.143865 | \n",
+ " 0.5867 | \n",
+ " 0.089775 | \n",
+ " 0.423900 | \n",
+ " 0.385721 | \n",
+ " 0.147728 | \n",
"
\n",
" \n",
"\n",
@@ -989,61 +1109,61 @@
],
"text/plain": [
" Data Algo K Train time (s) Predicting time (s) RMSE MAE \\\n",
- "1 100k als 10 6.6739 0.0716 0.963173 0.752521 \n",
- "2 100k svd 10 4.2722 0.2882 0.938681 0.742690 \n",
- "3 100k sar 10 0.3608 NaN NaN NaN \n",
- "4 100k ncf 10 63.8578 NaN NaN NaN \n",
- "5 100k fastai 10 96.2693 0.0533 0.943093 0.744332 \n",
- "6 100k bpr 10 6.6029 NaN NaN NaN \n",
- "7 100k bivae 10 10.4133 NaN NaN NaN \n",
- "8 100k lightgcn 10 14.7697 NaN NaN NaN \n",
- "9 1m als 10 3.3558 0.0262 0.860522 0.679683 \n",
- "10 1m svd 10 43.4842 3.4605 0.883017 0.695366 \n",
- "11 1m sar 10 3.3966 NaN NaN NaN \n",
- "12 1m ncf 10 818.8540 NaN NaN NaN \n",
- "13 1m fastai 10 800.6882 0.4382 0.874729 0.695722 \n",
- "14 1m bpr 10 70.1201 NaN NaN NaN \n",
- "15 1m bivae 10 157.1440 NaN NaN NaN \n",
- "16 1m lightgcn 10 417.1986 NaN NaN NaN \n",
+ "1 100k als 10 6.8526 0.0587 0.962863 0.747088 \n",
+ "2 100k svd 10 4.0902 0.2698 0.938681 0.742690 \n",
+ "3 100k sar 10 0.3344 NaN NaN NaN \n",
+ "4 100k ncf 10 66.8339 NaN NaN NaN \n",
+ "5 100k fastai 10 69.7469 0.0338 0.942754 0.745138 \n",
+ "6 100k bpr 10 5.8205 NaN NaN NaN \n",
+ "7 100k bivae 10 11.4762 NaN NaN NaN \n",
+ "8 100k lightgcn 10 16.2290 NaN NaN NaN \n",
+ "9 1m als 10 7.0365 0.0355 0.858791 0.677568 \n",
+ "10 1m svd 10 41.6351 2.8386 0.883017 0.695366 \n",
+ "11 1m sar 10 3.2292 NaN NaN NaN \n",
+ "12 1m ncf 10 816.1049 NaN NaN NaN \n",
+ "13 1m fastai 10 663.2788 0.3985 0.874907 0.695758 \n",
+ "14 1m bpr 10 66.0371 NaN NaN NaN \n",
+ "15 1m bivae 10 152.0912 NaN NaN NaN \n",
+ "16 1m lightgcn 10 463.8591 NaN NaN NaN \n",
"\n",
" R2 Explained Variance Recommending time (s) MAP nDCG@k \\\n",
- "1 0.258453 0.254536 0.0910 0.004091 0.041346 \n",
- "2 0.291967 0.291971 14.9461 0.012873 0.095930 \n",
- "3 NaN NaN 0.0853 0.113028 0.388321 \n",
- "4 NaN NaN 4.8731 0.094349 0.369862 \n",
- "5 0.285296 0.287658 3.2034 0.025590 0.148119 \n",
- "6 NaN NaN 1.5408 0.132478 0.441997 \n",
- "7 NaN NaN 1.5687 0.146126 0.475077 \n",
- "8 NaN NaN 0.0577 0.117371 0.407634 \n",
- "9 0.412199 0.406352 0.0547 0.002039 0.025166 \n",
- "10 0.374910 0.374911 210.9069 0.008828 0.089320 \n",
- "11 NaN NaN 2.1385 0.066214 0.313502 \n",
- "12 NaN NaN 52.5052 0.063974 0.348130 \n",
- "13 0.386589 0.389092 54.6923 0.026299 0.183996 \n",
- "14 NaN NaN 30.3135 0.083546 0.390911 \n",
- "15 NaN NaN 29.6179 0.092682 0.433709 \n",
- "16 NaN NaN 0.5851 0.087846 0.416899 \n",
+ "1 0.258405 0.255016 0.0782 0.004697 0.046619 \n",
+ "2 0.291967 0.291971 13.8704 0.012873 0.095930 \n",
+ "3 NaN NaN 0.0836 0.113028 0.388321 \n",
+ "4 NaN NaN 3.5393 0.108609 0.398754 \n",
+ "5 0.285810 0.288468 2.7415 0.025896 0.151481 \n",
+ "6 NaN NaN 1.9365 0.132478 0.441997 \n",
+ "7 NaN NaN 1.4382 0.146126 0.475077 \n",
+ "8 NaN NaN 0.0451 0.121633 0.417629 \n",
+ "9 0.413262 0.408737 0.0491 0.002683 0.030447 \n",
+ "10 0.374910 0.374911 190.6115 0.008828 0.089320 \n",
+ "11 NaN NaN 1.9796 0.066214 0.313502 \n",
+ "12 NaN NaN 48.9872 0.065931 0.357964 \n",
+ "13 0.386338 0.388787 47.2290 0.026237 0.184787 \n",
+ "14 NaN NaN 27.4882 0.083546 0.390911 \n",
+ "15 NaN NaN 27.5727 0.094632 0.438106 \n",
+ "16 NaN NaN 0.5867 0.089775 0.423900 \n",
"\n",
" Precision@k Recall@k \n",
- "1 0.045387 0.015172 \n",
+ "1 0.049629 0.016688 \n",
"2 0.091198 0.032783 \n",
"3 0.333828 0.183179 \n",
- "4 0.330117 0.165953 \n",
- "5 0.130541 0.053903 \n",
+ "4 0.349735 0.181576 \n",
+ "5 0.131813 0.054491 \n",
"6 0.388229 0.212522 \n",
"7 0.411771 0.219145 \n",
- "8 0.352916 0.190530 \n",
- "9 0.031373 0.009830 \n",
+ "8 0.360976 0.196052 \n",
+ "9 0.036707 0.011461 \n",
"10 0.082856 0.021582 \n",
"11 0.279692 0.111135 \n",
- "12 0.318900 0.109476 \n",
- "13 0.167550 0.055682 \n",
+ "12 0.327249 0.111665 \n",
+ "13 0.168494 0.056014 \n",
"14 0.357777 0.142510 \n",
- "15 0.396058 0.148564 \n",
- "16 0.378532 0.143865 "
+ "15 0.399039 0.149889 \n",
+ "16 0.385721 0.147728 "
]
},
- "execution_count": 16,
+ "execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
@@ -1055,9 +1175,9 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python (reco_full)",
+ "display_name": "reco",
"language": "python",
- "name": "reco_full"
+ "name": "conda-env-reco-py"
},
"language_info": {
"codemirror_mode": {
@@ -1069,7 +1189,12 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.6.11"
+ "version": "3.7.13"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "5659d8898e613821d32ce6048323094401019f14577fd202084feb69010a20de"
+ }
}
},
"nbformat": 4,
diff --git a/recommenders/datasets/criteo.py b/recommenders/datasets/criteo.py
index 9801ad4a11..7b639938bf 100644
--- a/recommenders/datasets/criteo.py
+++ b/recommenders/datasets/criteo.py
@@ -157,7 +157,26 @@ def extract_criteo(size, compressed_file, path=None):
extracted_dir = path
with tarfile.open(compressed_file) as tar:
- tar.extractall(extracted_dir)
+ def is_within_directory(directory, target):
+
+ abs_directory = os.path.abspath(directory)
+ abs_target = os.path.abspath(target)
+
+ prefix = os.path.commonprefix([abs_directory, abs_target])
+
+ return prefix == abs_directory
+
+ def safe_extract(tar, path=".", members=None, *, numeric_owner=False):
+
+ for member in tar.getmembers():
+ member_path = os.path.join(path, member.name)
+ if not is_within_directory(path, member_path):
+ raise Exception("Attempted Path Traversal in Tar File")
+
+ tar.extractall(path, members, numeric_owner=numeric_owner)
+
+
+ safe_extract(tar, extracted_dir)
filename_selector = {"sample": "dac_sample.txt", "full": "train.txt"}
return os.path.join(extracted_dir, filename_selector[size])
diff --git a/recommenders/evaluation/python_evaluation.py b/recommenders/evaluation/python_evaluation.py
index a762fa10bd..3e972bc7e1 100644
--- a/recommenders/evaluation/python_evaluation.py
+++ b/recommenders/evaluation/python_evaluation.py
@@ -420,11 +420,11 @@ def precision_at_k(
rating_pred,
col_user=DEFAULT_USER_COL,
col_item=DEFAULT_ITEM_COL,
- col_rating=DEFAULT_RATING_COL,
col_prediction=DEFAULT_PREDICTION_COL,
relevancy_method="top_k",
k=DEFAULT_K,
threshold=DEFAULT_THRESHOLD,
+ **kwargs
):
"""Precision at K.
@@ -450,7 +450,7 @@ def precision_at_k(
Returns:
float: precision at k (min=0, max=1)
"""
-
+ col_rating = _get_rating_column(relevancy_method, **kwargs)
df_hit, df_hit_count, n_users = merge_ranking_true_pred(
rating_true=rating_true,
rating_pred=rating_pred,
@@ -474,11 +474,11 @@ def recall_at_k(
rating_pred,
col_user=DEFAULT_USER_COL,
col_item=DEFAULT_ITEM_COL,
- col_rating=DEFAULT_RATING_COL,
col_prediction=DEFAULT_PREDICTION_COL,
relevancy_method="top_k",
k=DEFAULT_K,
threshold=DEFAULT_THRESHOLD,
+ **kwargs
):
"""Recall at K.
@@ -498,7 +498,7 @@ def recall_at_k(
float: recall at k (min=0, max=1). The maximum value is 1 even when fewer than
k items exist for a user in rating_true.
"""
-
+ col_rating = _get_rating_column(relevancy_method, **kwargs)
df_hit, df_hit_count, n_users = merge_ranking_true_pred(
rating_true=rating_true,
rating_pred=rating_pred,
@@ -522,13 +522,13 @@ def ndcg_at_k(
rating_pred,
col_user=DEFAULT_USER_COL,
col_item=DEFAULT_ITEM_COL,
- col_rating=DEFAULT_RATING_COL,
col_prediction=DEFAULT_PREDICTION_COL,
relevancy_method="top_k",
k=DEFAULT_K,
threshold=DEFAULT_THRESHOLD,
score_type="binary",
discfun_type="loge",
+ **kwargs
):
"""Normalized Discounted Cumulative Gain (nDCG).
@@ -553,7 +553,7 @@ def ndcg_at_k(
Returns:
float: nDCG at k (min=0, max=1).
"""
-
+ col_rating = _get_rating_column(relevancy_method, **kwargs)
df_hit, _, _ = merge_ranking_true_pred(
rating_true=rating_true,
rating_pred=rating_pred,
@@ -621,11 +621,11 @@ def map_at_k(
rating_pred,
col_user=DEFAULT_USER_COL,
col_item=DEFAULT_ITEM_COL,
- col_rating=DEFAULT_RATING_COL,
col_prediction=DEFAULT_PREDICTION_COL,
relevancy_method="top_k",
k=DEFAULT_K,
threshold=DEFAULT_THRESHOLD,
+ **kwargs
):
"""Mean Average Precision at k
@@ -657,7 +657,7 @@ def map_at_k(
Returns:
float: MAP at k (min=0, max=1).
"""
-
+ col_rating = _get_rating_column(relevancy_method, **kwargs)
df_hit, df_hit_count, n_users = merge_ranking_true_pred(
rating_true=rating_true,
rating_pred=rating_pred,
@@ -736,6 +736,26 @@ def get_top_k_items(
}
+def _get_rating_column(relevancy_method: str, **kwargs) -> str:
+ r"""Helper utility to simplify the arguments of eval metrics
+ Attemtps to address https://github.com/microsoft/recommenders/issues/1737.
+
+ Args:
+ relevancy_method (str): method for determining relevancy ['top_k', 'by_threshold', None]. None means that the
+ top k items are directly provided, so there is no need to compute the relevancy operation.
+
+ Returns:
+ str: rating column name.
+ """
+ if relevancy_method != "top_k":
+ if "col_rating" not in kwargs:
+ raise ValueError("Expected an argument `col_rating` but wasn't found.")
+ col_rating = kwargs.get("col_rating")
+ else:
+ col_rating = kwargs.get("col_rating", DEFAULT_RATING_COL)
+ return col_rating
+
+
# diversity metrics
def _check_column_dtypes_diversity_serendipity(func):
"""Checks columns of DataFrame inputs
diff --git a/recommenders/utils/gpu_utils.py b/recommenders/utils/gpu_utils.py
index 8bfc2602b3..c428e96493 100644
--- a/recommenders/utils/gpu_utils.py
+++ b/recommenders/utils/gpu_utils.py
@@ -17,13 +17,18 @@
def get_number_gpus():
"""Get the number of GPUs in the system.
-
Returns:
int: Number of GPUs.
"""
try:
- return len(cuda.gpus)
- except CudaSupportError:
+ import torch
+ return torch.cuda.device_count()
+ except (ImportError, ModuleNotFoundError):
+ pass
+ try:
+ import numba
+ return len(numba.cuda.gpus)
+ except Exception: # numba.cuda.cudadrv.error.CudaSupportError:
return 0
@@ -61,37 +66,45 @@ def clear_memory_all_gpus():
logger.info("No CUDA available")
-def get_cuda_version(unix_path=DEFAULT_CUDA_PATH_LINUX):
- """Get CUDA version.
-
- Args:
- unix_path (str): Path to CUDA version file in Linux/Mac.
-
+def get_cuda_version():
+ """Get CUDA version
+
Returns:
str: Version of the library.
"""
- if sys.platform == "win32":
- raise NotImplementedError("Implement this!")
- elif sys.platform in ["linux", "darwin"]:
- if os.path.isfile(unix_path):
- with open(unix_path, "r") as f:
+ try:
+ import torch
+ return torch.version.cuda
+ except (ImportError, ModuleNotFoundError):
+ path = ""
+ if sys.platform == "win32":
+ candidate = (
+ "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\version.txt"
+ )
+ path_list = glob.glob(candidate)
+ if path_list:
+ path = path_list[0]
+ elif sys.platform == "linux" or sys.platform == "darwin":
+ path = "/usr/local/cuda/version.txt"
+ else:
+ raise ValueError("Not in Windows, Linux or Mac")
+
+ if os.path.isfile(path):
+ with open(path, "r") as f:
data = f.read().replace("\n", "")
return data
else:
- return "No CUDA in this machine"
- else:
- raise ValueError("Not in Windows, Linux or Mac")
+ return "Cannot find CUDA in this machine"
def get_cudnn_version():
- """Get the CuDNN version.
-
+ """Get the CuDNN version
+
Returns:
str: Version of the library.
-
"""
- def find_cudnn_in_headers(candidates):
+ def find_cudnn_in_headers(candiates):
for c in candidates:
file = glob.glob(c)
if file:
@@ -111,21 +124,23 @@ def find_cudnn_in_headers(candidates):
else:
return "Cannot find CUDNN version"
else:
- return "No CUDNN in this machine"
-
- if sys.platform == "win32":
- candidates = [
- "C:\\NVIDIA\\cuda\\include\\cudnn.h",
- "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\include\\cudnn.h",
- ]
- elif sys.platform == "linux":
- candidates = [
- "/usr/include/x86_64-linux-gnu/cudnn_v*.h",
- "/usr/local/cuda/include/cudnn.h",
- "/usr/include/cudnn.h",
- ]
- elif sys.platform == "darwin":
- candidates = ["/usr/local/cuda/include/cudnn.h", "/usr/include/cudnn.h"]
- else:
- raise ValueError("Not in Windows, Linux or Mac")
- return find_cudnn_in_headers(candidates)
+ return "Cannot find CUDNN version"
+
+ try:
+ import torch
+ return torch.backends.cudnn.version()
+ except (ImportError, ModuleNotFoundError):
+ if sys.platform == "win32":
+ candidates = [r"C:\NVIDIA\cuda\include\cudnn.h"]
+ elif sys.platform == "linux":
+ candidates = [
+ "/usr/include/cudnn_version.h",
+ "/usr/include/x86_64-linux-gnu/cudnn_v[0-99].h",
+ "/usr/local/cuda/include/cudnn.h",
+ "/usr/include/cudnn.h",
+ ]
+ elif sys.platform == "darwin":
+ candidates = ["/usr/local/cuda/include/cudnn.h", "/usr/include/cudnn.h"]
+ else:
+ raise ValueError("Not in Windows, Linux or Mac")
+ return find_cudnn_in_headers(candidates)
diff --git a/setup.py b/setup.py
index 79a13ecedb..e99e4e82da 100644
--- a/setup.py
+++ b/setup.py
@@ -74,7 +74,7 @@
"spark": [
"databricks_cli>=0.8.6,<1",
"pyarrow>=0.12.1,<7.0.0",
- "pyspark>=2.4.5,<4.0.0",
+ "pyspark>=2.4.5,<3.3.0",
],
"dev": [
"black>=18.6b4,<21",