FAQ

Q: Do participants only submit their results for the Phase 2 Questions?

A: All we need is the prototypicality ratings for the pairs in Phase1Answers-CATEGORY.txt. The script scale_to_maxdiff.pl will convert the ratings to answers for the Phase2Questions-CATEGORY.txt. The assumption here is that your algorithm must be generating a rating scale internally. That is, you wouldn't answer the MaxDiff questions directly, without going through an intermediate step of generating a rating scale somehow. Let us know if this assumption is incorrect.

Q: If we only need to assign prototypicality ratings for the pairs in Phase1Answers-CATEGORY.txt, then why are the Phase 2 Questions included in the download package?

A: The reason for including Phase2Questions is that you might want to tune your algorithm to maximize its score on the Phase2Questions (score_maxdiff.pl), instead of the Spearman correlation score (score_scale.pl). One strategy would be to submit two sets of results, one set for an algorithm that maximizes score_maxdiff.pl on the training data and another set for an algorithm that maximizes score_scale.pl on the training data. (Ideally the same algorithm would maximize both, but that's one of the questions we hope to answer by having the two scores.)

Q: Must the results be on a scale of 0 to 100 (like the example file) or can any numeric scale be used?

A: Any scale can be used. It won't matter to the scoring scripts (score_scale.pl and score_maxdiff.pl).

Q: Must the result file submitted be in sorted order (like the example file)?

A: The scoring scripts don't care about order.

Q: Is there any limit on the number of approaches a team may submit?

A: Three.