What is the meaning of sas in the results?

[chgao96]

I am really confused about that. Hope for a response.

[rdevooght]

Hi,

Are you talking about the "sps@10" ? Otherwise I don't see what you mean by "sas".
The sps@k is a measure that we designed (this is why you haven't seen it yet), which is the percentage of user for which the method is able to predict the first item in the test set (given k trials).
You can see it as a recall@k where only the first item in the test set is considered to be a positive result.

The goal of the sps is to measure whether the method is able to predict the next item that a user will interact with. This is indeed what we measure, because for each user the items are sorted by the timestamp of the user-item interactions, and therefore the first item of the test set is the next item with which the user interacted.

Is it clearer ?

[chgao96]

Thank you for a response.
Yeah, it is "sps". I made a spelling mistake.
Well, I understand the meaning of "sps". But, I have never seen a published paper taking it as a metric. So could you offer any reference? Thanks a lot.
Besides, what is the full name of "sps"?

[rdevooght]

It stands for short-term prediction success.
This is actually a measure we came up with because we wanted to measure how well can the model predict the next user interaction, which is something never measured in recommender system research.
I am pretty sure that this measure exists under another name somewhere in the sequence prediction field, since it is such a simple measure, but I could not tell you where.

We could have used the perplexity instead, which is widely used to evaluate language model.
It would have been more familiar to those coming from the sequence prediction field, but I believe that the sps is more intuitive for those coming from the recommender system field, which is why we chose it.