Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare for cross validation-based benchmarking #60

Merged
merged 19 commits into from
Apr 3, 2022
Merged

Prepare for cross validation-based benchmarking #60

merged 19 commits into from
Apr 3, 2022

Commits on Mar 21, 2022

  1. Disallow n_folds<2, which returns no training data

    @takuti
    takuti committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    645d10e View commit details
    Browse the repository at this point in the history

  2. Logging cross validation results

    @takuti
    takuti committed Mar 21, 2022
    Configuration menu
    Copy the full SHA
    fb6988a View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2022

  1. Rename function args: u, i -> user, item

    @takuti
    takuti committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    16bf8e1 View commit details
    Browse the repository at this point in the history

  2. Rename variable: k -> topk

    @takuti
    takuti committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    e8312a5 View commit details
    Browse the repository at this point in the history

  3. Allow ranking metrics to take topk=nothing 

    In this case, use `length(pred)` as a recommendation size.
    @takuti
    takuti committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    4b09626 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2022

  1. Evaluate accuracy only on truth samples 

    `ealuate()` unnecessarily made predictions for all user-item pairs.
Comparison must be done between truth vs. pred.
    @takuti
    takuti committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    c2cecb4 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2022

  1. Pass topk for ranking evaluation

    @takuti
    takuti committed Apr 2, 2022
    Configuration menu
    Copy the full SHA
    314701d View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2022

  1. Remove unnecessary sort in evaluation for ranking metric 

    Returned list of item-score tuples from `recommend` is already sorted by
the scores.
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    70bab16 View commit details
    Browse the repository at this point in the history

  2. Use sortperm to get ranked list of truth items

    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    ee03e17 View commit details
    Browse the repository at this point in the history

  3. Make recommendation only for unobserved items 

    when a recommender is evaluated by a ranking metric.
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    07e6573 View commit details
    Browse the repository at this point in the history

  4. Drop unobserved items from truth for ranking evaluation 

    `truth` must be a ranked list of observed items for correct evaluation.
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    54f6747 View commit details
    Browse the repository at this point in the history

  5. Avoid topk to be more than recommendation size

    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    b66673b View commit details
    Browse the repository at this point in the history

  6. Treak test criteria so cross validation can pass as expected 

    Cross validation has some randomness, and it may or may not return very
poor/good result.
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    83e0c0d View commit details
    Browse the repository at this point in the history

  7. Skip recommendation and measurement when truth is empty 

    Adjust cross validation test cases to increase the probability of seeing
an empty `truth` list.
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    71082ff View commit details
    Browse the repository at this point in the history

  8. Explicitly support leave-one-out cross validation 

    If `n` equals to the number of all samples, `n`-fold CV is same as
LOOCV.
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    2b6e9cf View commit details
    Browse the repository at this point in the history

  9. Support multi-threadhing for ranking evaluation 

    Top-k recommendation for every single user is costly. It'd be
recommended to parallelize whenever possible.
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    45025aa View commit details
    Browse the repository at this point in the history

  10. Add leave-one-out CV as a synonym of k-fold CV

    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    4cd0ced View commit details
    Browse the repository at this point in the history

  11. Refactor with k-fold splitter for DataAccessor

    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    1ec68b7 View commit details
    Browse the repository at this point in the history

  12. Remove accuracy=NaN check as it's prevented at upstream 

    by checking the size of test samples
    @takuti
    takuti committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    79a757a View commit details
    Browse the repository at this point in the history