Dear Rose,
Congratulations! On behalf of the Program Committee, we are delighted to inform
you that your submission:
Paper#: 9
Title: Personalized Recommendations using Knowledge Graphs: A Probabilistic Logic Programming Approach
has been accepted by the RecSys 2016 conference as a long paper for oral
presentation.
As usual, RecSys is a very competitive forum. This year, RecSys received a
total of 294 submissions and accepted 29 long papers (acceptance rate 18.2%)
and 22 short papers (acceptance rate 20%). The review process was extremely
selective due to the high number of submissions. The Program Committee worked
very hard to ensure that every paper got at least three reviews. All papers,
particularly those at the borderline, were also reviewed and thoroughly
discussed by a Senior Program Committee member and by the PC Chairs.
The reviews for your paper are attached to this e-mail. Please carefully read
them and revise your paper accordingly. Please also proofread your paper before
submission to check for typos and grammatical errors.
A. The instructions for camera-ready-copy preparation are available at:
http://www.sheridanprinting.com/typedept/recsys.htm
Kindly review the templates and formatting guidelines. Please familiarize
yourselves with the formatting and review in general for the helpful formatting
hints included.
B. The submission deadline for the submission of your camera-ready version is
***July 10th, 2016***.
No extensions will be given. Kindly attend to your submission early if you have
travel planned near the submission deadline. Papers and abstracts not submitted
on time will be dropped from the proceedings.
C. You will receive two new emails shortly:
C1. One email from ACM Rightsreview (rightsreview@acm.org) with a link to the
electronic ACM copyright-permission form(s) to be completed. Upon completing
the electronic form, you will receive a confirmation email. Contained within
the confirmation email will be the ACM copyright-permission block text,
conference data, and DOI string/url specific for your submission and mandatory
to appear on the first page. See formatting help in item #1 on how to include
this text and minimize the space impact on the first page.
C2. One email from Sheridan Communications (acm@sheridanprinting.com). This
email will contain your assigned submission id#, link to the formatting
instructions (same as item A above), and a unique-link to submit your final
publication ready version on or before the submission deadline.
Registration for the conference is now open -
https://recsys.acm.org/recsys16/registration. Let us kindly remind you that at
least one author for each accepted paper is required to register by July 10th
in order for the paper to be included in the proceedings.
Thank you again for submitting to RecSys 2016 and we look forward to seeing you
and your co-authors at the RecSys 2016 conference in Boston!
Best regards,
Jill Freyne and Pablo Castells
ACM RecSys 2016 Program Chairs
----------------------- REVIEW 1 ---------------------
PAPER: 9
TITLE: Personalized Recommendations using Knowledge Graphs: A Probabilistic Logic Programming Approach
AUTHORS: Rose Catherine and William Cohen
OVERALL EVALUATION: 0 (borderline paper)
REVIEWER'S CONFIDENCE: 4 (high)
Relevance for RecSys: 4 (good)
Novelty: 2 (poor)
Technical quality: 3 (fair)
Significance: 3 (fair)
Presentation and readability: 4 (good)
----------- Review -----------
This paper presents three content based methods to recommend items for users using external knowledge graphs (KGs). Based on a general-purpose probabilistic logic system called ProPPR, the author manually creates several rules to define the similarity, relatedness, links and other relationships among different entities w.r.t. the recommending items. For the three methods, the EntitySim method depends on several learning rules and TypeSim is simple yet effective, while GraphLF combines latent factor learning and item similarity calculation together and is the most complex. The author shows that the performance of these KG-based methods is much better than state-of-the-art method HeteRec_p. Generally speaking, the relative advantages among the three proposed methods depends on the density of the recommending dataset.
I have some questions as below:
I think the motivation of the KG-based recommendation as described in your Abstraction and Introduction is not strong enough. Specifically, you do not claim the improvements against the general social recommender systems in your settings, especially in the cold start scenario.
- In Sec. 3.3, how to extract features for edges?
- In Sec. 3.5, how to quantise the predictability? And also, please show the convergence of your learning method.
- In Sec. 3.6, what’s the meaning of “many dimensions D along which the values of both X and Y are high”? What meaning does the “high value” present?
- In experiments, what knowledge graph do you use? What are the details of the characteristics for the using KG? For your experiments comparing to HeteRec_p, did you use the same KG as that?
Typos:
- In Sec. 2.1, “dom(R_i + 1)” should be “dom(R_{i+1})”.
- In Sec. 4.5, “higher than the density at y=2”, I think it should be “k=2”.
- In Sec. 4.5, what is “worse than 633%”? I think it should be “worse <100% than”.
Things I like:
- clear writing
- detailed described related works
- unified running example
Things I dislike:
- simply described learning detail, such as feature engineering, optimisation methods
- lack for description of KGs
- too much man-craft rules
- less novelty
Generally speaking, it is a borderline work.
----------------------- REVIEW 2 ---------------------
PAPER: 9
TITLE: Personalized Recommendations using Knowledge Graphs: A Probabilistic Logic Programming Approach
AUTHORS: Rose Catherine and William Cohen
OVERALL EVALUATION: 1 (weak accept)
REVIEWER'S CONFIDENCE: 4 (high)
Relevance for RecSys: 4 (good)
Novelty: 3 (fair)
Technical quality: 4 (good)
Significance: 4 (good)
Presentation and readability: 4 (good)
----------- Review -----------
This paper focuses on the creation of very advanced recommendation algorithms that exploit facts from knowledge graphs. The unique aspect is that the algorithm is specified declaratively as a probabilistic logic program, which gives high flexibility; the results demonstrate how this can be useful when different types of data are brought together to improve personalization.
A drawback of the paper is that it has to assume some familiarity with ProPPR, or at least leave out some of the details of the mechanisms in ProPPR that actually make the whole approach feasible and effective. However, if we assume that the inference procedure is correct and efficient, given that it is already published in more specialized venues to check these aspects, then the current paper is a good showcase on how to carry out recommendation with such an approach. T
he different approaches evaluated can be expressed elegantly, and the experimental results confirm intuition: that knowledge base data is more important if the amount of usage data available for training is low. This finding that sparseness level determines the method that is best suited could naturally be expected, but it is not well reported in the current literature; and this work shows it clearly.
I did not like the presentation of the experimental data, where indeed the results improve with an increasing k, but do not report to what extent the amount of data considered shrink (and thereby naturally increase the probability of a better MRR - there are less competing data points that could "overtake" the target in the ranking). With the newer 20% of reviews as targets and k=100, how difficult is it to find a target, and would that smaller test collection not explain the improvements in MRR, instead of the difference in sparsity? Also, the argument to not need to compare to collab filtering approaches becomes invalid, as [26] does not give the results on these constructed sets.
Overall, I am enthusiastic about the method, and the flexibility offered to integrate various data sources to improve recommendation, and think the paper provides a nice enough showcase, in spite of the minor concerns with the presentation of the experimental results.
Minor:
In 4.3, you write that NB is expected to do very badly, but the text earlier suggests that NB would do well; you may want to make this more internally consistent.
----------------------- REVIEW 3 ---------------------
PAPER: 9
TITLE: Personalized Recommendations using Knowledge Graphs: A Probabilistic Logic Programming Approach
AUTHORS: Rose Catherine and William Cohen
OVERALL EVALUATION: 3 (strong accept)
REVIEWER'S CONFIDENCE: 3 (medium)
Relevance for RecSys: 5 (excellent)
Novelty: 4 (good)
Technical quality: 4 (good)
Significance: 4 (good)
Presentation and readability: 5 (excellent)
----------- Review -----------
The authors demonstrate the utility of knowledge graph methods in recommending movie titles. They consider both class-aware and class-agnostic entity models, as well as comparing their methods to a state-of-the-art method and a simple naive bayes method. The approach combines content-based and collaborative filtering, demonstrating that sparse data are treated much better with the knowledge graph methods, which is of considerable utility for cold-start scenarios, including the recommendation of newly released items.
I recommend acceptance of this paper and consideration for best paper for the following reasons:
- the approach addresses a common and meaningful problem in generating recommendations in real-world datasets that often suffer from sparseness, but where a detailed knowledge graph is available.
- the algorithms proposed are thoughtful models for the problem, including comparisons of explicit entity classes as well as SVM features that capture latent meaning in entity relationships. The algorithms are also efficient extensions of a random walk approach similar to page rank.
- The evaluation of the approach was thorough and well controlled, demonstrating an improvement over both competing and a reasonable baseline method (NB)
- I appreciate the characterization and limitations of when these sophisticated algorithms are required or even advantageous, as the author demonstrate that dense datasets are treated better with the simplistic naive bayes approach.
- the paper is well organized and well written
I have no major complaints with the work, and I would love to see more approaches with this level of consideration in modeling the data inherently in the recommendation algorithm.