Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write new Experiment or ExperimentAnalysis code module #371

Open
mdekstrand opened this issue Apr 10, 2024 · 0 comments
Open

Write new Experiment or ExperimentAnalysis code module #371

mdekstrand opened this issue Apr 10, 2024 · 0 comments

Comments

@mdekstrand
Copy link
Member

mdekstrand commented Apr 10, 2024

Right now RecListAnalysis is good but limited — only computes per-user metrics.

It would help standardization of evaluation procedures if we had a more coherent "analyze" (and maybe "run") tool for experiments. The first version, of course, would just be for analysis.

  • Specify experiment axes instead of inferring them?
  • Support global metrics
  • Specify list lengths as analysis parameter
  • Support metrics with additional data (novelty, etc.)
  • Clean up metric interface design
  • Support analysis (sig tests, CIs, distributions, etc.)
  • Support results in DuckDB?

This ticket is really probably its own epic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

No branches or pull requests

1 participant