Kam Hin Ho

Phrase 1

SVD and SVD++

SVD and SVD++ are ways of factorizing a matrix into three other matrices (A = UΣVᵀ). It runs principal component analysis on both the users and the items, and returns the matrices we need. It is known to produce a very accurate result, and it was used in the Netflix prize competition. It has some interesting algebraic properties and conveys important geometrical and theoretical insights about linear transformations. In the video, we introduced a biased SVD algorithm and the unbiased SVD algorithm. The SVD++ is the extension of SVD, and it takes into account the implicit ratings. An implicit rating describes the fact that a user rated an item j regardless of the rating value. It abstract each user with a factors vector and has a lower RMSE than SVD.

Limitation of SVD and SVD++

Essentially, SVD and SVD++ are projecting each user and item onto a latent space represented by a latent vector. The more similar the two latent vectors are, the more related the corresponding users' preference. We can measure the similarity of any two latent vectors with cosine-similarity or dot product. However, the dot product limits the expressiveness of user and item latent vectors.

NCF

With the limitation of the SVD and SVD++, NCF overcomes the limitation by using its neural architecture for learning the interaction function from data.

1. Input Layer binarises a sparse vector for a user and item identification where:

   • Item (I): 1 means the user u has interacted with Item(i)

   • User (u): To identify the user

2. The embedding layer is a fully connected layer that projects the sparse representation to a dense vector. The obtained user/item embeddings are the latent user/item vectors.

3. Neural CF layers use Multi-layered neural architecture to map the latent vectors to prediction scores. The embedded user and item vectors are concatenated before passing through a series of fully connected layers, which maps the concatenated embeddings into a prediction vector as output.

4. The final output layer returns the probable class using Logistic Sigmoid Function.

As mentioned above, we are exploring the performance and accuracy of SVD, SVD++, and NCF in collaborative filtering. Both SVD and SVD++ models will be evaluated using Root Mean Squared Error (RMSE). We are using a library called Surprise to train and test the model. We used the movie lens data with nearly 5.5 million rows of data on training and testing the SVD, SVD++. We used 80 % of the data to train both models and 20% to test them, cross-validation was performed to prevent overfitting. The results showed that SVD has an average of 0.8126 for RMSE, and SVD++ has an average of 0.8620. In other words, the SVD performs better than SVD++ in terms of RMSE (Lower RMSE). In terms of Performance, SVD took an average of 287.42 seconds to train and 16.35 seconds to test the model, while SVD++ takes an average of 424.99 seconds to train and 8.04 to test.

For NCF, we used the pytorch_lightning library as our deep learning framework. Since we require negative samples to indicate movies that the user has not interacted with, we have generated 4 negative samples for each row of data. The ratio of negative to positive samples is 4:1. Then, we fed the user input vector and item input vector to the user embedding and item embedding respectively, which results in smaller, denser user and item vectors. The embedded user and item vectors are concatenated before passing through a series of fully connected layers, which maps the concatenated embeddings into a prediction vector as output. Finally, we apply a Logistic Sigmoid function to obtain the most probable class. In terms of performance, NCF took the longest to train and fit. (30 mins to train, and 11 mins to test) However, it has the best accuracy out of all 3 models (0.90 in accuracy).

Although there is some trade-off in performance and accuracy, it is recommended to use NCF to perform collaborative filtering if the resources are available. We expect SVD++ performs better as it takes into account the data implicit feedback. However, our result shows that the SVD actually has a better RMSE than SVD++ in addition to a shorter train time.