This is very interesting work. I do have a couple of questions for you though.
Tiny Eval
101

Hello!

  1. It would be easier to sort the references with only one score for recommendations purposes. For example, if you have a vector of probability of ratings for (5,4,3,2,1) how do you compare (0.50,0.1,0.1,0.1,0.2) and (0.30,0.7,0.,0.,0.) ? (Unsure the model can actually output theses cases). You would have to draw some rules here to have an 1 dimensional score. Taking the max proba class and sort based on that can be one choice. It’s not obvious that it could bring more performance in the recommender. A good question is then how do you compare these models. It’s not really possible offline.
  2. I don’t have the code and output anymore so I won’t create this graph. Would have been nice to show I concur.

Thanks for reading!