[Feature] Support string output path in EvalAlgorithmInterface.evaluate(save="...")

[athewsey]

Today the save parameter of EvalAlgorithmInterface.evaluate() is just a boolean.

From hunting around I found that the output path is taken either from an environment variable or else a default under /tmp... And that it should also be overrideable by setting the obviously-supposed-to-be-private property eval_algo._eval_results_path.

IMO it's harder and less obvious than it should be for a developer using the library to save results in a folder they want. It'd be much easier if we could support eval_algo.evaluate(save="my/cool/folder") and ideally automatically create the provided folder if it doesn't already exist?

[keerthanvasist]

Thanks for bringing this up @athewsey! What you said seems to make absolute sense. We will take another look at this decision, why it was done this way, and if there isn't one, we will update it soon.

[athewsey]

Any update on this? Would be useful to be able to actually save evaluation results where we want

[keerthanvasist]

This is in our backlog. We are currently evaluating different proposals. We will update this soon.